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A MULTIPLE GROUP METHOD OF FACTORING THE 
CORRELATION MATRIX* 


L. L. THURSTONE 
THE UNIVERSITY OF CHICAGO 


There are a number of methods of factoring the correlation 
matrix which require the calculation of a table of residual correla- 
ticns after each factor has been extracted. This is perhaps the most 
laborious part of factoring. The method to be described here avoids 
the computation of residuals after each factor has been computed. 
Since the method turns on the selection of a set of constellations or 
clusters of test vectors, it will be called a multiple group method of 
factoring. The method can be used for extracting one factor at a 
time if that is desired but it will be considered here for the more in- 
teresting case in which a number of constellations are selected from 
the correlation matrix at the start. The result of this method of 
factoring is a factor matrix F' which satisfies the fundamental re- 
lation FF’ = R. 


Selection of clusters 

This method starts with the selection of a set of clusters of tests 
which are significantly correlated as in the original attempt to deter- 
mine the dimensionality of a correlation matrix. It is assumed here 
that an estimate has been made for the diagonals. The grouping can 
be done systematically by listing all of the tests with high correla- 
tions in Any given column of the correlation matrix. The rest of the 
square table for the group so selected is then completed from the cor- 
relation matrix. Those tests are eliminated from the group which 
have any low correlations so that the group which remains shows 
only significant correlations. If there are any negative signs, we re- 
verse the sign of one or more variables so that the resulting corre- 
lations for the group are all positive. Then we have a cluster of tests 
whose vectors can be assumed to be roughly collinear in the sense 
that the test vectors of each group are contained in a cone with fairly 
small angular spread. 

When one cluster has been selected, the next cluster may begin 
with a test which is not included in the first cluster. One proceeds in 
the same manner for each successive group. When all of the test vec- 

* This study is one of a series of investigations in the development of multiple 
factor analysis and application to the study of primary mental abilities. We wish 
to acknowledge the financial assistance from the Social Science Research Com- 


mittee of The University of Chicago which has made possible the work of the Psy- 
chometric Laboratory. 
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tors have been included in one or more clusters, it may be desirable 


to eliminate each test from all but one of the clusters. This should be - 


the cluster in which the test has the highest average correlation. It 
is not desirable to do this grouping with any hair-splitting computa- 
tions because clusters are rarely unique. For purposes of ultimate 
interpretation the clusters are frequently misleading because several 
clusters may be found in the test vectors of a single hyperplane. How- 
ever, the dimensionality can be roughly estimated from the number 
of clusters that can be distinguished in the given correlation coeffici- 
ents. The present purpose of arranging the tests in clusters is mere- 
ly to define an oblique set of unit reference vectors which are linearly 
independent and which span the common factor space or as much of 
it as can be seen by inspection of the given correlations. 


Computational procedure 

In Table 1 we have an illustrative example of nine variables and 
their correlations. Inspection of this table enables us to select three 
groups denoted s, , s. , and s, with several tests in each group. In this 
example there are three tests in each group, but in practice it is not 
necessary to have the same number of tests in each group. One starts 
the factoring by selecting the groups that seem to be conspicuous and 
there is no need to be concerned about whether the selection is in any 
sense unique. In general one should avoid assigning a test to two 
groups but that is not prohibited. Nor is it essential that all of the 
tests be represented in the subgroups although that was done in the 
present example. 

In the matrix S we have three rows corresponding to the three 
groups. The tests in each group are listed in the table. In row S$, we 
write the sum of the correlations for the tests in group s, . Since there 
are three tests in this group, the row s, will have the sums of rows 
2, 5, and 9 of the correlation matrix. The correlation matrix has now 
been augmented by an additional row, which does not alter its rank 
because this row is a linear combination of the given rows of the cor- 
relation matrix. If an additional column were added on the right side 
of the correlation matrix in the same manner, we should have an aug- 
mented correlation matrix of the same rank. This has not been done 
because it would be simply the transpose of the rows of matrix S 
which have already been computed. The row s, is computed in the 
same manner by summing rows 4, 6, and 7 of the correlation matrix. 
The row s,; is computed by summing rows 1, 3, and 8 of the correla- 
tion matrix. These are the three groups into which the test battery 
has been divided by inspection of the correlation matrix. 

The sum of the corresponding entries in each row of the matrix 
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S are then computed and recorded as shown in matrix 7. The entry 
4.930, for example, is the sum of the entries in row s, and in columns 
2, 5, and 9. The other sums are determined in the same manner. This 
square matrix T of order 3 X 3 represents the lower right corner of 
the augmented correlation matrix in which the rank is unaltered. 


TABLE 1 
1 2 3 4 5 6 7 8 9 I II III 
1 65 .00 .49 28 24 66 86 68 28 1 .234 612 577 
2 00 .64 .24 82 .48 .00 00 .00 56 2 .757 -.244 -.091 
3 49 24 58 12 .18 42 00 .68 .49 $8 .410 .047 .640 
4 28 82 12 65 .66 .42 .68 .00 .28 4 .567 .580 -.215 
5 24 48 18 .66 .72 86 .54 .00 42 5 .780 .376 -.215 
6 66: 00. .42 42 36 72. 64. 24 6 270: 678 
36 .00 .00 .63 .54 .54 .81 .00 .00 7% .243 .838 -.218 
8 68 .00 .68 .00 .00 .54 .00 .81 86 8 .162 .178 .867 
9 28 .56 .49 .28 .42 24 00 386 .65 9 .734-.134 .306 
Rx F 
83 
$, 52 1.68 .91 1.26 1.62 .60 .54 .86 1.63 4.930 2.400 1.790 
85 1.30 .82 .54 1.70 1.56 1.68 1.98 .54 .52 2.400 5.360 2.380 
LTT 241.70 40 2 1.62 86-207 1.18 1.790 2.380 5.540 
S T 
8) 85 S83 
?P; .234 .757 .410 .567 .730 .270 .243 .162 .734 2.220 1.081 .806 
Po .562 .138 .233 .734 .674 .726 .855 .233 .225 1.037 2.315 1.028 
PD, -752 .102 .722 .170 .178 .688 .153 .879 .480 -760 1.011 2.354 
Vv U 
8, : Tests 2,5,9 p, 1.000 467 342 
8, : Tests 4,6,7 467 1.000 .437 
8, : Tests 1,3,8 p, .342 437 1.000 
I 
Pp, 1.000 0 
p, 467 0 
p, .342 814 .886 
I 
Pp, 1.000 -.529 -.199 
p, 1181 —401 
0 0 1.129 
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The next step is to compute the reciprocals of the square roots 
of the diagonal elements of the matrix 7. These are the weights w, , 
wz, and w;, which are shown to the right of the matrix T. 

The matrix V’ is computed by applying the weight w, to the first 
row of S , the weight w. to the second row of S, and so on. Similarly, 
the same weights on the rows of 7 give the square matrix U as shown. 
The matrix V’ and U may be thought of as three rows by which the 
correlation matrix could be augmented instead of the rows of S and 
T. The matrix U would then be the lower right corner of the aug- 
mented correlation matrix in which the rank is still unaltered. 

The three weights w,, w., and w; are then applied to the col- 
umns of the matrix U and we then have the matrix R,, with unit diag- 
onals. The calculations are in part checked by the unit diagonals of 
this matrix. This matrix shows the cosines of the angular separations 
of the three unit reference, vectors that have been chosen. Each one 
of them contains the centroid of a cluster of test vectors. 

If we write the transpose of the matrix V’ we have an oblique 
factor matrix V as in previous methods of rotation from the orthogo- 
nal factor matrix F', but we have obtained here the oblique matrix 
directly and without computing the orthogonal factor matrix F which 
is usually obtained first by the various factoring methods. 

Since we have here an oblique factor matrix V as well as the co- 
sines of the angular separations of the unit reference vectors for that 
factor matrix V , we can determine the transformation which should 
carry the oblique factor matrix V into an orthogonal factor matrix F . 
Since the location of an orthogonal frame in the system defined by V 
and U is not unique, we must first define in some convenient way the 
location of the desired orthogonal reference axes of F'. We shall do 
this by locating the first of the orthogonal reference axes collinear 
with the first of the oblique axes. Then the first column of F' will be 
the same as the first column of V. The next orthogonal reference 
axis will be located in the plane of the first two oblique axes and or- 
thogonal to the first one. This process is in fact the diagonal method 
of factoring applied to the correlation matrix R,,. When this is car- 
ried out routinely, we have the factor matrix F'n = A'm, in which the 
three oblique reference axes p are defined in the orthogonal frame m . 

The inverse of the transpose of this matrix gives An 7) = (F’pm)7. 
We can now write the transformation by which the orthogonal factor 
matrix F can be written for the whole test battery. The matrix Fyn is 
then the additional three rows of F' which describe the oblique refer- 
ence axes p. This transformation is 


VA(=F (1) 


from which we have 
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FA=V (2) 


where A is already numerically given. 

Applying this transformation to the matrix V by equation (1) 
we obtain the factor matrix F', which is shown in the upper right 
corner of the table. 

When this factor matrix has been written, a residual table can 


be computed for the three factors to determine whether additional 


factors need to be extracted. In the present example, which is exact, 
the residuals vanish identically. In a practical problem with experi- 
mental data, the residual table should be computed and inspected to 


determine whether additional factors should be extracted. If that 


should be desirable, one proceeds with the residual table as with the 
original correlation matrix in the present example. Clusters are then 
listed from the residual correlation matrix until the tests with sig- 
nificant residuals seem to be represented. The process is repeated so 
as to add one or more columns to the factor matrix F. 

In the present case we have one set of residuals to compute for 
the entire factor matrix F instead of three sets of residuals as would 


have been required in most of the current methods of factoring the . 


correlation matrix. It will also be seen that the order of the inverse 
to be computed is equal to that of the number of factors determined 


by this process. For example, if seven factors are to be determined - 


and if the process starts with seven linearly independent clusters, 
then there is an inverse of order 7 X 7 to be computed. If the factor- 
ing proceeds with, say, only four clusters, then there is an inverse of 
order 4X 4 to be computed and the fourth factor residuals should then 
reveal three additional clusters resulting in three columns to be added 
to the factor matrix F. The. computational labor is probably mini- 
mized by finding as many clusters in the given correlation matrix as 
there are common factor dimensions so that only one table of cor- 
relational residuals need be computed. However, if the first estimate 
of the number of clusters is too small, one merely repeats the process 
until the residuals vanish. If the factoring is done in several steps, 
the inverses are of course of smaller order. 

The method of selecting groups for the present example was the 
method used in the early development of multiple factor analysis. It 
serves the purpose of separating the variables into a number of lin- 


early independent groups that can be used for factoring as has been | 


shown here. This method of group selection depends on the simple 


principle that those tests should be put in the same group which are 


highly correlated. If all of the test vectors were of equal length, i.e., 
if the communalities were all the same, then this principle would give 
a set of tests in each group that are roughly collinear. Since all of the 
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test combinations were not examined, it is to be expected that the 
groupings are not necessarily the best that could have been found. 

The purpose of the grouping of the tests is to find sets that are 
roughly collinear. An improved method of accomplishing this result 
is to plot pairs of columns of the correlation matrix. Those tests 
which give roughly linear plots through the origin are nearly col- 
linear and they should be placed in the same group. This method of 
grouping the tests is an improvement because it does not exclude a 
test of low communality from the group to which it should belong. 

In using this method on the present nine-variable example, we 
start with the test whose column in the correlation matrix has the 
highest absolute sum. That is test 6. Plotting column 6 with the other 
columns we find that 1 and 6 are nearly collinear. These are placed in 
the first group. The next highest absolute sum is that of column 5. 
Plots with other columns show that 4 and 5 are nearly collinear. Elimi- 
nating these four tests, the next highest absolute sum is for test 9. 
Examining the plots for rough approximation to collinearity one could 
select the following groups: a) 1, 6, 8; b) 4, 5, 7; c) 2, 3,9. As has 
been pointed out before, the grouping of tests is not unique. This 
grouping obtained by the graphical method could have been used in 
the present example and the result would have been the same because 
this example is exactly of rank 3. In dealing with experimental data 
it is best to select a number of groups that is larger than the probable 
number of common factors that can be interpreted. This insures that 
there will be at least one residual factor. This would not be success- 
ful on fictitious data of exact rank as in the present case because one 
would then find that the correlation matrix R,, would not yield an in- 
verse. Such a situation is rare with experimental data where it is 
preferable to have one or more residual common factors to be left 
without interpretation. The purpose of the residual factors is to in- 
sure that the common factor space has been covered with a sufficient 
number of dimensions of significant: variance for interpretation. 

In practice the grouping can be done by combining those tests 
which have roughly proportional columns of coefficients as judged by 
inspection. The graphs merely make the proportionality more obvious. 
Further, it is not necessary that all of the tests in each group be col- 
linear or even nearly so. All that is required is that the whole corre- 
lation matrix be divided into sections that are linearly independent. 
This is accomplished by combining those tests which are significantly 


correlated. 
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THE RELATION OF ITEM DIFFICULTY AND INTER-ITEM 
CORRELATION TO TEST VARIANCE AND RELIABILITY* 


HAROLD GULLIKSEN 
COLLEGE ENTRANCE EXAMINATION BOARDt 


; Under assumptions that will hold for the usual test situation, it 
is proved that test reliability and variance increase (a) as the aver- 
age inter-item correlation increases, and (b) as the variance of the 
item difficulty distribution decreases. As the average item variance 
increases, the test variance will increase, but the test reliability 
will not be affected. (it is noted that as the average item variance 
increases, the average item difficulty ge goed .50). In this devel- 
opment, no account is taken of the effect of chance success, or the 
possible effect on student attitude of different item difficulty distribu- 
tions. In order to maximize the reliability and variance of a test, 
the items should have high intercorrelations, all items should be of 
the same difficulty level, and the level should be as near to 50% as 


possible. 


The following proofs are restricted to cases where the test is 
composed of a number of items, each of which is counted as one point 
if correct and as zero if incorrect, and the raw score is the number of 
correct answers. The influence of chance success, and the possible 
effect on student attitude of various distributions of item difficulty 
are not considered. The effect of number of items on test parameters 
will not be pointed out, since this factor has been adequately dealt 
with by previous writers on test theory (see 4, 5, and 6). 

Consider the relation of the item difficulty distribution to the 
maximum possible correlation between the item and total test score. 
In order to determine this relationship, we shall begin with the rela- 
tion of item difficulty to the maximum correlation between two items. 
If N designates the number of persons taking the test, and p; and py 
represent the proportions of persons answering items i and j cor- 
rectly (py; = p;), then the sum of the cross-products of the scores on 
items i and 7 for the N individuals cannot exceed Np;. Applying the 
usual gross score formula for correlation and simplifying we get 


pi (1 — pj) 


*The desirability of determining this relationship has been indicated Ths 
previous writers. Work on the present paper arose out of some problems rai 
by Dr. Herbert S. Conrad in connection with an analysis of aptitude tests. 

+ On leave for Government war research from the Psychology Department, 


University of Chicago. 
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where 7;;"** is the maximum possible correlation between items i and 
j. Formula (1) or its equivalent is also given in reference (1), for- 
mula 19, and in reference (3), formula 3. Only when p; = p; can 
r;j™* equal unity. The maximum possible correlation declines very 
sharply as the difference between p; and p; increases. Figure 1 shows 


1.0 


7) 


2 3 5 6 7 


FIGURE 1 
Combinations of p, and p, for which the Maximum Correlation is 1.0, 0.7, 


0.5, 0.3, 0.1, and 0.0. 


the difficulty combinations for which the maximum correlation is 
1.00, .70, .50, .30, .10, and .00. In order to emphasize the sharpness 
of the drop in maximum inter-item correlation with increase in dif- 
ficulty difference between two items, Figure 2 shows the highest 
possible correlation between two items for various values of p, and 
px. for the special case in which p, + ~—1. As the difficulty differ- 
ence between the two items increases, the drop in maximum correla- 
tion is very sharp. For example, an item answered correctly by 40% 
of the group cannot possibly correlate more than .667 with an item 
answered correctly by 60% of the group. 

We can see from Figures 1 and 2 that only if all items are of the 
same difficulty value is it possible to have perfect correlation among 
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| 
0.0 
>, 0.0 od 3 5 7 9 1.0 
1.0 8 7 5 4 3 2 0.0 


FIGURE 2 
Maximum Inter-Item Correlation for Two Items of Difficulty p, and p, for 


the Special Case p, + p, = 1.00. 


the items. Also, if the difficulty value is near 50%, then a difference 
in difficulty has less effect on 7;;™* than if the difficulty value is far- 
ther from 50%. It should be emphasized that this maximum is a 
theoretical one. It would be desirable to have some evidence for vari- 
ous types of items to show how great a difficulty difference it takes 
to become practically serious in decreasing inter-item correlation. 
We may now see how the item difficulty distribution affects the 
maximum possible item-total correlation. Applying the formula for 
the correlation of a sum to a test of K items, we have 
K 
9; 
(ri =1) (2) 


Ty 
Oj Of 


where 
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ri, is the item-total correlation, 

o; is the standard deviation of the test, and 

o; and o; are the standard deviations of items i and j, 
respectively. 


We also note that 
where p; is the proportion answering item i correctly, and qi; =1 — pi. 
Let us arrange the items in difficulty order such that 
Di > D2 Pi > > De. 


Then substituting (1) and (3) in (2) and simplifying we obtain 


i K 
PIGt+a DV; 
j=. 
1 = : (4) 
orVDi Gi 


where r;,"** is used to designate the maximum possibile item-total 
correlation. Formula (4) above corresponds to formula 17 in refer- 
ence (1) and formula 4 in reference (3). By arranging the items in 
difficulty order and running cumulative totals beginning at q, for the 
q’s and at px for the p’s, a routine for calculating (4) for each item 
of any given set of items can be set up. It can be seen that if all item 
difficulties (p; and y;) are equal, then all r;,"* are equal, and that 
the more the p; differ from each other, the greater will be the spread 
in values of 7;,™*. 


We may summarize the foregoing material in 


Theorem A: The correlation between two items can be 
as high as unity only when they are of the same difficulty; 
as the difficulty difference between two items increases, the 
maximum possible correlation between them drops sharply. 


Let us turn now to the effect of item difficulty on the mean, vari- 
ance, and reliability of the total test score. 
Since the mean of a sum equals the sum of the means of the sepa- 
rate parts, we may write 
x 
M,=Xpi=Kp, (5) 


t=1 


where 


= 
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M,=the mean raw score, 
K =the number of items in the test, 
; =the proportion of persons answering the i-th item correctly. 


p =the average of the p;; 


[see also reference (1), formula 2, and reference (6), formula 22]. 
We may express the standard deviation of the raw score distribu- 
tion by using the usual formula for the variance of a sum given in 
reference (5), page 213, formula 163. .Applying this formula, we - 
May write 
x. = 
=> ij aig;, (ri =1) (6) 


j=1 
where 
VDi Gi : (7) 
In these equations 


o,° = the variance of the distribution of raw scores, 

7,; = the correlation between items 7 and j, 

pi; —the proportion of persons answering the i-th item correctly, 
K =the number of items in the test. 


Formula (6) shows that the raw score variance will increase with 
each increase in correlation between any two items and will increase 
as the difficulty value of any item approaches .50. 

For a given average item difficulty level, how do changes in the 
variance of the item difficulty distribution affect raw score variance? 
This relationship can be observed most easily if the form of formula 
(6) is changed. Let us use 7,00). to designate the correlation between 
the intercorrelation of two items, and the product of their standard 
deviations. Some of the assumptions in the development which fol- 
lows will be more reasonable if we omit from the correlation all 
terms which include 7;; = 1. This means summing the terms in the 
matrix of item variances and covariances, excluding the terms lying 
on the principal diagonal. We will designate this procedure by indi- 
cating the sum over (K? — K) terms instead of over K? terms. 

In order to indicate the sum over the (K? — K) non-diagonal 
terms, let us adopt the definition 


K?2-K K K K 
So? (where r;;= 1). 
ix} 


i=1 j=1 F 
Using this definition and applying the usual formula for correlation 
gives 


| 
| 


PSYCHOMETRIKA 


K*-K 
> rij 0; — (K? — K) rij 6; 
4,j=1 
— ’ (8) 
(K? K)o,, 


where 


Ti; is the average inter-item correlation, (excluding 7;; = 1) 


0; 0; is the average of the products of item standard devia- 
tions for all possible combinations such that i ~ j, and 


the other symbols have the same definition as in equations 
(6) and (7). 


From (8) and (6) we have 
K*-K K 
4,j=1 
(9) 
= (K? — K) [7 Or, oo + + K(o?), 
where 


(o*) is the average item variance, and the 
other symbols have the same definitions as in previous equations. 
To eliminate the term o; o; we adopt the definition 


K?-K K K 
4,j=1 je1 i=1 


and note that 
K 2 


i=1 
Substituting N times the mean for the “sum of scores” gives 
(K? — K) 0; 0; + K (o*) = K*(a)?. 
Then multiplying both sides by 7i; , we have 
(K* — K) 0; + K(o*) = 14; K*(o)*, (10) 
where 


(o*) is the average item variance, and is not to be confused 
with 

(oc)? which is the square of the average item standard devia- 
tion. 


HAROLD GULLIKSEN 85 
We may substitute (10) in (9) and get 
= (Kk? = K)?,,00) Or Goo + i; K?(o)? K (0?) Tis), (11) 


where 
K = the number of items in the test, 


Teo) = the correlation of the correlation between two items with the 
product of their standard deviations (including only the non- 
diagonal elements in the correlation matrix) 


o, = the standard deviation of the distribution of inter-item correla- 
tions (not including the terms 7;; = 1) 


7;; = the average inter-item correlation (i * j) 


ooo = the standard deviation of the products of the item standard 
deviations (i # 7) 


(oc)? = the square of the average item standard deviation 
(oc?) = the average item variance. 


Let us consider the probable magnitude of these terms. As far 
as is known no direct evidence on the value of 7,,cc, has ever been ob- 
tained, so that setting an upper bound for it is a guess. However, 
since there is an attempt to make 7;; as large as possible, it should 
have a small range in a well-constructed test, which would make for 
a low value of 7,,c0). Let us assume that in a well-constructed test 
Y,.e0) Will not be much greater than 0.1. If we estimate that rj; will 
seldom be higher than .8 and follow the policy of discarding items for 
which r;; < 0.1, then 0.1 seems a plausible guess for the value of Or, 5+ 


The maximum value of oo is .25, and if there are very few items be- 
yond the 10,90 difficulty level, the lower value of oo is 0.09. With a 
range of 0.16, it seems that 0.1 is a reasonable upper bound for ooo. 
By making correspondingly reasonable approximations, we may say 


that ri;, (o)?, and (o*) will usually lie between 0.1 and 0.2. Since 
ror,» and o,e0) Will each be small, their product will be still 


r 


smaller, and hence the first term of (11) can usually be neglected. 
Neglecting the first term, we can see from equation (11) that the raw 
score variance of a test will increase with an increase in the number 
of items, the average inter-item correlation, the average item stand- 
ard deviation, or the average item variance. 

The effect of altering the dispersion of item difficulty while hold- 
ing the average item difficulty constant will depend on the behavior of 


4 
> 
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(o*) and (c)?. First investigating (oc) we see that o;2 = p; — p;? and 
hence 


=p — (p*). (12) 


We may denote the variance of the item difficulty distribution by o,? 
and from the usual formula for variance write 


= (p*) — (p)*, (13) 


where 


(p?) is the average of the squared item difficulties, and (p)? is the 
square of the average item difficulty. If we substitute (13) in (12) 
we find that 


(0?) =p — (p)? — (14) 


from which it is clear that for a given average item difficulty the 
average item variance (c”) decreases as the variance of the item dif- 
ficulty distribution increases. This means that as far as the last term 
of equation (11) is concerned, the test variance will increase as the 
variance of the item difficulty distribution decreases. 

No simple method has been found for determining the variation 
of (c)? with variation in o, for a given p. It would seem reasonable, 
however, to find that (c)? varied in the same general way as (o”). A 
rigorous proof of this fact can be given by writing 


oi = — pi) = (.5 + d;) (.5 — di) 
(15) 


= y (.25 — d?;) = (.25 — 


In equation (15) d; is defined as p; — .5, and x; = d;*. The last ex- 
pression in (15) may be expanded in a McLauren series, giving 
2-1 27-1-3 2?-1-3-5 
2 


o; — 0.5 2 -— 2? - x 
2! 3! 4! 


(16) 


5! n! 


Applying Cauchy’s ratio test for convergence we find that this series 
converges for —1/4 < x < + 1/4, which means that the series con- 
verges for 0 < p < 1. Summing terms like those in (16) and substi- 
tuting d* for x , we have 


| 

4 
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i=1 i=1 im 
(17) 
x x 
4-1 i=1 
the first three terms of which may be rewritten as 
‘tis A K A 
de? de? 
i=1 
(18) 


K 
—4pdee— 


#=1 i=1 


where p =p — 0.50 and e; = p: — p. Referring to equation (18) and 
the second term of equation (11), we see that for a given average 
item difficulty, the second term of equation (11) will increase as the 
dispersion of item difficulty decreases.* Thus we see that whenever 
the first term of equation (11) can be neglected, equations (14) and 
(18) show that for a given average item difficulty the test variance 
will increase as the variance of the item difficulty distribution de- 
creases. In reference (4) it is stated, without proof, that test vari- 
ance increases as item difficulty variance decreases. 
We may summarize the foregoing results on test variance in 


Theorem B: 


For any test'in which >> the raw score 
variance will increase as 


a, the average inter-item correlation increases, 


b. the average item standard deviation or variance in- 
creases (e.g., as the average item difficulty ap- 
proaches .50) 


c. the item difficulty variance decreases for any given 
average item difficulty 


It may be pointed out that the condition under which Theorem B is 
true will hold for any well-constructed test. 

Let us now consider the effect of item difficulty on the reliability 
of the test. From formula (11), together with the Kuder-Richardson 
formula (2) given in reference (6), we may write 


* It can be shogun be a fairly lengthy argument (which is omitted here) that 
the effect of negative odd moments can be neglected since they are small in com- 


parison with the even moments. 
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o,? — K (0?) oi? 
R= (19) 


where 


R= the reliability of the total test, 
=the reliability of item 7, and 


the other terms have the same meanings as in equation (11). If we 
designate the correlation between item reliability and item variance 
by r_,and follow a procedure similar to that followed in equations 


(7) to (11) we obtain 
o,2 — K(o*) + + K 7; (0?) 
R= (20) 


where 

7, is the average item reliability, 

ts is the correlation between item reliability and item variance, 

o is the standard deviation of the distribution of item reliabilities, 
o. ‘is the standard deviation of the distribution of item variances, and 
the other terms have the same meanings as in equation (11). Again, 


assuming that we have a well-constructed test, it is reasonable to sup- 
pose that r a a and a will each be small, hence their prod- 


r(aa) ro? 


ucts will be still smaller, and terms involving these products may be 
neglected. If we neglect such terms, equation (20) may be rewritten 
and rearranged as 

() (1-7 

R=1-— (o*) ( — (21) 
(0?) + — (07)] 

This equation shows that test reliability increases either with average 
item reliability or average inter-item correlation. To show the effect 

of item difficulty, (21) may be rearranged as 

1 


The effect of the term | —— 
(0?) 


Jean be seen by noting that oo? =(0”) — (ec)? 


| 
| 
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approaches a maximum value of 1.0 as the 


Thus the ratio 


variation in item difficulty approaches zero. Hence it can be seen 
that, other things being equal, test reliability increases as the 
variance of the item standard deviations decreases. This result is 
also given in reference (2) for the special case in which all inter- 
item correlations are unity. 

It is to be noted that the variance of the item standard deviations 
is lowered by concentrating items either around one difficulty level, 
or around any two difficulty levels p; and »; such that p; + p; = 1.00. 
For example, the variance of item standard deviation is zero if half 
the items are of 30% and half of 70% difficulty. However, as we have 
seen from Theorem A, inter-item correlations (and hence test reli- 
ability) can increase markedly as item difficulty differences decrease. 
Therefore, in order to increase test reliability, it is probably better 
to lower the variance of item standard deviations by concentrating 
all items about one difficulty level than by concentrating items about 
two difficulty levels whose sum is unity.* 

It should be noted, however, that the effect (shown in formula 
22) of the variance of item standard deviation upon test reliability 
is slight. For example, if half of the items had a difficulty value of .5 
and the other half a difficulty value of .9, then the value of the ratio 


2 
Gy? would equal 16/17, which differs but slightly from unity. It is 


to be noted, nevertheless, that if all items are concentrated about one 
difficulty level, then this slight effect reinforces the marked effect 
noted in Theorem A above; hence it seems clear that for a given aver- 
age item difficulty, one is likely to get the best reliability by having 


all items of that particular difficulty. 
The relationship shown in equations (21) and (22) may be stated 


qualitatively in 
Theorem C: Fora testinwhichr 
are negligible, the test reliability will increase with 
a. am increase in average item reliability, 
b. an increase in average inter-item correlation, 
c. a decrease in the variance of the item standard devi- 
ations. 


i 


*In reference (4) the increase in reliability with decrease in item difficulty 
variance is indicated as contingent upon the increase in test variance, and not 
upon the decrease in variance of item standard deviations. 


{ 2 
| (o 
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It may again be pointed out that the condition under which Theorem 
C is true will hold for any well-constructed test. 

As a guide to test construction, the material contained in Theo- 
rems A, B, and C may be summarized in the following statement. If 
we ignore the effect of chance success, and assume that we are dealing 
with a “well-constructed” test, then in order to maximize the reli- 
ability and variance of the test, items should have high intercorrela- 
tions, all items should be of the same difficulty level, and this level 
should be as near to 50% as possible. 

One point may be noted where the results implicit in Theorems 
A and C [or in formulas (1) and (22)] are at variance with current 
test practice. These formulas indicate that if all the items are con- 
centrated at one difficulty level, (even though that be a very high 
level of difficulty such as 20%, or a very low level of difficulty such as 
80%) test reliability can be higher than is possible if the items cover 
a rather wide difficulty range, such as 10% to 50%, or 90% to 50%. 
Current practice in test construction favors spreading the items over 
a difficulty range rather than concentrating all items at one difficulty 
level. Current practice would be particularly opposed to concentrat- 
ing all items at a very high or at a very low level of difficulty. Since 
the theory presented here does not allow for the influence of chance 
success, it does not allow for the fact that very difficult items would 
have a relatively much higher proportion of chance successes, and 
hence would necessarily have very low inter-item correlations, re- 
sulting in low test reliability. Also it may be that a test composed en- 
tirely of very difficult items would have an adverse effect upon stu- 
dent attitudes, and hence upon test reliability. However, neither of 
these considerations apply to a test composed entirely of very easy 
items. Whether it is actually best to concentrate all items at one dif- 
ficulty level, in accordance with the theory presented here, or to dis- 
tribute items over a difficulty range in accordance with present test 
practice, can be determined only by experiments such as those report- 
ed in (7), (8), and (11). 

It may be, if items of graded difficulty levels are used, that count- 
ing one point for each item correct is not a proper scoring method. 
The score assigned should rather be a best estimate of the difficulty 
level reached, analogous to that used in the Binet test. Such a method 
of test scoring has been refined and applied in reference (10) ; how- 
ever, the problem has not been generally solved for objective tests. 

Another limitation in the theory here developed should be pointed 
out. The criterion of maximizing test variance cannot be pushed to 
extremes. Test variance is a maximum if half of the population 
makes zero scores, and the other half makes perfect scores. Such a 
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score distribution is not desirable for obvious reasons, yet current 
test theory provides no rationale for rejecting such a score distribu- 
tion. Obviously the “best” test score distribution is one which accu- 
rately reflects the “true” ability distribution in the group, but there 
is perhaps little hope of obtaining such a distribution by the current 
procedure of assigning a score based upon sheer number of correct 
answers. At present the only solution to such difficulties seems to lie 
in some type of absolute scaling theory (9), to replace the “number 


correct” score. 


10. 


11. 
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_I.B.M. tabulating equipment can be of considerable help in re- 
ducing the time and increasing the accuracy of multiple factor analy- 
sis, even if used for only a part of the calculations. Once the plug- 
board is wired and those cards punched which are used over and over 
again, problems involving any number of variables can be handled 
with dispatch. The correlation matrix is listed, the totals verified, 
and the signs changed on the tabulator. Then the factors and the 
residual coefficients are calculated by means of a calculator. Tuck- 
er’s procedure has been modified by using a calculator instead of a 
multiplying punch, by reducing the number of cards used, by sim- 
plifying checks on calculations, by simplifying plugboard wiring, and 

y preparing work sheets on tabulator paper. Extraction of factors 
from 24 variables at the rate of one in four hour’s time seems to 
justify the use of the tabulating equipment on small problems. 


Multiple factor analysis is a valuable statistical technique, but 
the amount of work involved is almost prohibitive unless machine 
methods are employed. Tucker’s procedure} demands the use of an 
80-counter tabulator and a multiplying punch, two pieces of equip- 
ment available only in the larger tabulating installations. Our adap- 
tation utilizes a 56-counter tabulator and a Monroe calculator Model 
AA-1, with negative multiplication and keyboard lock devices, and 
makes possible the calculation of factor problems involving over 100 
variables with considerably less work than if the problem were per- 
formed on a calculating machine only. 

This report consists of three parts, (1), a brief discussion of the 
fundamental factor theorems and equations, (II), the procedures for 
machine calculations of the factor loadings, and (III), an appendix 
summarizing instructions for each tabulation. 


I. The Factor Analysis Problem 
Suppose that 20 tests are given to 100 individuals. It is probable 
that the scores made by any individual are determined by his abilities, 
such as intelligence, manual dexterity, aptitude in numerical work, 
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etc. It is conceivable that the score made by a given individual on 
any test could be predicted if the amount of his abilities pertinent to 
the test were known. This implies, of course, that the relating for- 
mula would be known also. Unfortunately, we seldom know all these 
facts. Consequently we must proceed on the theory that the correla- 
tion between the scores on the different tests is influenced by the abil- 
ities possessed by the individual tested. 

If the same ability or abilities are required by each of two tests, 
then scores on these two tests will tend to vary together. Further- 
more, as the number of common abilities increases and as the rela- 
tive importance of these abilities in the two tests becomes the same, 
the more perfect will be the relationship between the scores made on 
the two tests. 

Realizing these facts we can now state the problem of factor 
analysis as a simple question. Knowing the test scores only, can we 
reverse the process given above and discover the abilities involved 
and their importance in each of the tests? It may broaden our con- 
cept if we speak of the abilities or common causes as factors. 

In a table of inter-correlation coefficients between test scores, it 
is often observed that certain groups of variables show relatively 
high inter-correlations with all variables within the group, and rela- 
tively lower ones between variables of the group and those outside. 
From the factor analysis viewpoint this situation would lead to the 
hypothesis that at least one factor is common to the tests within the 
group. Other groups unrelated to the first suggest the presence of 
other factors. Some members of a given group might belong to other 
groups through the medium of common factors. The factor analysis 
technique attempts to clarify this grouping phenomenon by measur- 
ing the tendency of each variable to join a cluster with other vari- 
ables because of the presence of common factors. 

1. Assumptions and Notation. Let K denote the number of vari- 
ables to be analyzed, as K tests, for example. Let N individuals be 
scored X,, X2,-°:: , Xx on these variables. Denote by 7;; the correla- 
tion between variables X; and X;. There will be (K + 1) /2 coeffici- 
ents of correlation. Assume each variable is made up of certain fac- 
tors as follows: 


(1) anerror factor due to imperfection in the instrument 

(2) specific factors present in that variable only (for simplic- 
ity we assume only one) 

(3) other factors present in varying amounts, each in at least 
one other variable in the group. For convenience of nota- 
tion we shall assume only three of these are present. This 
will be seen to involve no restriction on the method. 


4 
4 
{ 
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To simplify formulas, let us assume that scores on all factors and all 
variables have been normalized so that all means are zero and all 
variances are unity. The following notation will be used: 


Ei= score on the error factor of the ith variable, 
L,y,z= “ “ three factors of the ith variable, which 
occur in at least one other variable, 


Zi = score on the ith variable. 


It is assumed that the factors are such that Z; = aix + biy + ciz + 
d;(Si) + e;(Ei). The coefficients a;, b;, c;, di, ¢ are called factor 
loadings. They are relative weights applied to the additive factors, 
which are of course variable, and are used to determine a value of Z; 
from known values of the five factors in any individual instance. This 
assumption of additivity is a restriction on the factors similar in na- 
ture to the assumption of linearity of regression in multiple correla- 
tion. The essential difference is that the identity of the variables on 
the right side is known in regression, but is to be discovered in factor 
analysis. In both cases the coefficients must be determined. We also 
assume that all factors are independent in the sense of being uncor- 
related. 

We shall now give a brief description of the Thurstone centroid 
method of factor analysis. 


2. Fundamental Formulas. By normalizing the variables we 

made the following equations true: 
i=N | i=N i=N 

—>X?7=1; 
we 

Similarly, for all other factors and combinations of factors we 
made means to equal zero, variances to equal unity and correlations 
to equal zero. 

The correlation, 7:2. , between Z, and Z, is given by 


i +=N 
[des + + Gres +d (S1)s + 


X + bey; + + d,(S2); + (#2) 


If these expressions be multiplied we see that two types of terms are 
obtained ; those with squares of factors, and those with cross-products. 
By hypotheses made above, all sums of squares divided by N are one, 
and all cross-products divided by N are zero. Hence the expression 


reduces to 
= + bibs + 
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Thus the correlation between two variables is given by the sum of the 
products of their loadings on common factors, since other correlations 
would result in the same form. 

In a similar way the variance of Z, , denoted by <,?, is 


1 i=" 
DS + diy, + 2; + d,(S1); + i]? 


i=1 
=a’?+ ¢,?+d,? + e,?. 
Let h,?=4,? + + 


3. Communality. We call h,?, defined above, the communrality 
of variable Z,. It is that portion of the variance of variable Z, due 
to those factors common to Z, and one or more of the other variables, 
That is to say, we can think of it as that amount of the variance of 
Z, which can be explained in terms of variation in factors included 
in the other variables which are being studied. Stated negatively, 
communality is that portion of the variance not due to error and spe- 
cific factors. 

In practical work it is clear that the communalities will be un- 
known. They could be determined only if the factor loadings were 
known a priori. Hence they must be estimated. A number of methods 
might be given for this estimation, but only the simplest will be de- 
scribed. It will be noted in the derivation to follow that slight errors 
in such estimation will have little effect on the calculated loadings. 

An upper bound to the communality of a test would be its reli- 
ability or self-correlation. Existence of specific factors would make 
the communality less than the reliability. On the other hand, we would 
usually expect the communality of a variable to be larger than any 
correlation coefficient using that variable with any other, sign being 
disregarded. This is due to the possibility of some specific factors. In 
rare cases it might be smaller. We use this approximate lower bound 
for estimating the communality. For variable Z, we pick the numeric- 
ally largest correlation coefficient in which Z, is used. The communal- 
ity of Z, is estimated as being this coefficient with positive sign. 


4. Finding the Factor Loadings of the First Factor. Consider 
all correlation coefficients which involve Z,, together with the com- 
munality of Z,. Let r:, denote this communality. 

11, + b,b, + c,¢, 
T 12 = + + 
—a,0, + 6,b, + 


Tin + bbe + 


4 
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Adding these, we obtain 


g=1 j=l j=1 


Similarly, 
K K K 
DA +b SO, 6; 
j=1 j=1 j=1 
and 
K K K K 
> Ux 4; + bx > b; + Cx 
j=1 j=1 
If these be added, we obtain 


The double summation on the left includes each correlation twice and 
each communality once. The terms on the right are the squares of the 
sums of the factor loadings of the three factors common to two or 
more tests. 

Assume the factor loadings to be determined in an order such 
that the sum of the loadings of the first factor is a maximum. In the 


K K 
notation above we wish to maximize > a;. This will occur if 5 6; = 
K 
> c; = 0. Some doubt exists as to the wisdom of this assumption. 
j=1 
Other systems of factor analysis depend on different hypotheses. 
Since we are concerned only with the Thurstone system, no attempt 
will be made to criticize the basic theory. 
The result of the above hypothesis is 


$0), 


i=1 j= 


where 5 r denotes the double sum. From this we obtain 3 a= vSr. 
We assume the positive square root. This involves no loss of gen- 
erality. It merely assigns a direction of meaning to factor one, as 
possession of an ability or lack of this ability. 
Now 


K K 
> = 4; 4; 
jet 


| K 2 K 2 K 2 
EEru=( Se) + (30) + ( Be). 
j=1 j=l j=1 j=1 
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from equation (1), since 
K K 
j=1 
This gives 
K K 
j=1 j=1 
: 
Sa, 


j=1 


This is, then, a formula for the calculation of the loadings of the 
first factor. In other words, it says that the factor loading of a par- 
ticular variable is a fraction with numerator given by the sum of all 
correlations involving that variable, including its communality, and 
denominator given by the square root of the sum of all correlations 
and communalities, each correlation entering twice, each communal- 
ity once. Thus the numerator is the sum of a column of the correla- 
_ tion matrix, and the denominator is the square root of the sum of 

the column sums. 


5. Finding Residuals. Return to the formula for the correla- 
tion coefficient 7;; = a;a; + bib; + cic;. We now know the values of 
the first factor loadings, the 2’s. Now we can solve for that portion 
of the correlation not dependent on factor one. This is called the first 
residual, denoted by 7;;.. and given by the formula 


Tija — Aid; . 


The table showing these residuals is called the first residual table. 

If we express 7;;., in the form 7;;. = bib; + e;¢;, it is clear that 
the residuals are correlations between the variables after the effect 
of factor one has been removed. In this sense they parallel partial 
correlations. 


6. Necessity of Sign Change. We now have in effect a new cor- 
relation matrix ready for analysis similar to that applied above. Di- 
rect repetition of the described procedure is not possible, however, 
since each column sum, and therefore the grand sum, of the residual 
table will be zero, except for rounding errors. Our hypothesis made 


K K 
above that 5 b; = S c; = 0 forces this result: Thus the formula which 
j=1 


would parallel the one used above, namely, 
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K 

DM ija 

i=1 jel 
is meaningless since both numerator and denominator are zero. This 
indeterminant form is evaluated by an artifice of sign changing. 


7. Sign-Change Procedure. It is well known that changing the 
sign of a variable changes the sign of the correlation of that variable 
with another. It would also change the sign of factor loadings in the 
same way. We can change the column totals and grand total of the 
residual table by changing the signs of some of the variables. Con- 
sistent with our desire to have each factor removed to account for 
the largest possible portion of correlation, we wish to maximize col- 
umn totals. This result is approximated in the following way. Delete 
diagonal entries, i.e., the communalities, from the residual table. The 
column sums now differ from zero. Change the signs of those vari- 
ables which show negative column totals until the maximum positive 
total is obtained. While one cannot specify this process exactly, cer- 
tain rules are helpful. Change the variables in a sequence of steps, 
not all at once. Do this in order of magnitude, changing the largest 
one (or the largest ones) first. Subsequent changes may indicate that 
a previous choice was unwise. In this case it should be reflected back. 
More specifically we would wish to reflect back to the original sign 
any variable which changed to a negative total on reflection of some 
other variable in a later step. This process will be discussed in more 


detail in section 3. 


8. Estimating Residual Communalities. Although we have resi- 
dual communalities calculated in the same way as residual correla- 
tions, it seems better not to use them, but to make new estimates at 
this time. Since original communalities were merely approximations, 
relative errors in residual communalities conceivably might be quite 
large. It is probable that the absolute value of the highest first resi- 
dual in each column might be a more accurate figure. Hence this value 
is used. As was mentioned above, it is apparent that errors in com- 
munality estimates will have little effect on factor loadings. 


9. Finding Loadings on Factor Two. We are now ready to ap- 
ply to the residual matrix after sign change the same process for find- 
ing factor loadings on factor two as was used in finding factor one 
loadings from the original correlation matrix. From these loadings 
and the first residual table we proceed to other factors in the same 


i 
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way. It is now clear that our assumption that our matrix contained 
three factors was not a restriction on the method. 


10. Changing Signs in the Original Correlation Matrix. From 
the procedure explained above, we can see that it would be advanta- 
geous to apply the sign-change procedure to the original correlation 
matrix before removal of the first factor. This will enable the re- 
moval of the factor of maximum importance as defined by the meth- 
od used. 


11. Rotation of Axes. When we think of factors as dimensions 
in Euclidean space, each variable will determine a single point whose 
coordinates are the factor loadings on that variable. Thurstone uses 
a rotation of axes to obtain factors in which identification or nam- 
ing is facilitated. Since we are proposing no modifications in this 
procedure, we shall not discus the problems involved in rotation. There 
are, however, many interesting aspects of the theory of rotation and 
its application to specific problems. The authors hope to consider this 
in later work. 


II. Procedure for Machine Calculation 

The procedure outlined below inclules the steps necessary to ob- 
tain the factor loadings for the centroid matrix. 

1. The Work Sheets. In order to minimize errors and to facili- 
tate calculations, work sheets were prepared on the tabulator. A por- 
tion of one is shown as Figure 1.* The original correlation matrix 
and each residual table were first laid out on these work sheets and 
the cards were punched from them. These sheets were laid out in 
units of 8 X 8 cells because a 56-counter tabulator will accommodate 
only 8 six-digit variables plus the card identification columns. Con- 
tinuous tabulator paper will accommodate 3 such units per page. 

Our problem included 79 variables and consequently 80 X 80 be- 
came the table limits. The work sheets carried variable numbers 
across the top printed from type 8 cards and variable numbers down 
the left side printed from type 0 cards. (See Codes in Table 1). 

Because type 0, type 8, and type 7 cards may be used for any 
factor tabulation, sets of these cards to accommodate 100 variables 
were kept on file. We selected 80 type 0 cards and 80/8 sets of type 8 
cards. The 10 type 8 cards for section 1 were sorted ahead of the 80 
type 0 cirds on (79-78)+¢ and tabulated on 4-copy paper. Then the 
10 type 8 cards for section 2 were substituted for those for section 1 
and the puck was retabulated. This procedure continued until work 


* All figures and tables are segregated at the end of this article. 
+ The number in parenthesis refer to card columns. 
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sheets for 10 sections were prepared. The wiring diagram is shown 
as Figure 2. Specific instructions for each operation are summarized 
in the Appendix. 

One page of the work sheet upon which the correlation coeffici- 
ents have been entered is shown as Table 2. The variable numbers 
across the top were printed from the type 8 cards, and those down 
the side from the type 0 cards. 


1.1 The correlation matrix. The original correlation matrix was 
laid out by entering the correlation coefficients in their proper cells 
below the diagonal on one set of work sheets. The sum of the 79 co- 
efficients for each variable was obtained. Then in order to avoid hav- 
ing to deal with minus coefficients, unity was added to each coefficient 
and a new table was laid out with each coefficient entered in its proper 
cell on both sides of the diagonal. 

Thus a coefficient of .0334 became 01.0334 and one of —.0544 be- 
came 00.9456, its complement. All coefficients were punched from. 
these work sheets as 6 digits. The zeros to the left of the decimals 
were not recorded merely to save work, as is shown in units 2 and 
3 of Table 2. 

It was found convenient in checking the sign-change procedure 
to carry a blank variable with all cel] entries as 01.0000. We used 
variable 76 but would recommend that variable 1 be used in the fu- 
ture. If the number of variables is not a multiple of 8, the extra col- 
umns in the last section may be left blank but the last rows in all 
sections should be filled with values of 01.0000. 

The communalities are not used at this time, and consequently 
the values in the diagonal cells from 1 X 1 to 80 X 80 are 01.0000. 
The estimated communalities will be added on the factor calculation 
sheets (operation 4). The type 1 cards are punched from these work 
sheets according to the code shown in Table 1. 


2. Coding and punching. The color and corner cut of the eight 
different types of cards used are important to the mechanics of the 
operations. For the most part we retained the card number assigned 
by Tucker in order that reference may be made to his work. The 
table number (78-79) identifies all operations necessary to obtain a 
set of factor loadings. All cards and tabulation sheets from the origi- 
nal correlation matrix until the first factor loadings were calculated 
were identified as Table 1. Table 2 identified the residual table and 
the second sign-change operations, the calculation sheets, and all cards 
used to obtain the second factor loadings. The sample sheets shown 
in this paper are taken from Table 10 and consist of parts of the 
operations made in obtaining the tenth factor loadings. 


a 
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2.1 The type 0 cards. These cards are a serially numbered set 
carrying variable number in (1-3) and unit number in (78-79). They 
were used in preparing the work sheets described in operation 1. 


2.2 The type 1 cards. Type 1 cards were punched from the work 
sheets of 1 + r matrixes. The data in each row of each section were 
punched on separate cards in columns (11-58). Column 4 was sight- 
verified, then the cards were count-sorted on (3-1) to verify these. 
columns. 


2.21 Tabulating the column totals. In order to verify the cal- 
culation of the complements and the punching, the type 1 cards were 
tabulated. (Setup and operation of tabulator is described in the Ap- 
pendix.) Column totals thus obtained were verified with the totals 
obtained in operation 1.1. These totals were used to calculate the 
column totals for the following residual table as described in opera- 
tion 5.2 and to calculate one-half the sign change sum (operation 
2.31). 


2.22 Listing the correlation table. When the totals were veri- 
fied and the cards corrected, the pack was counter-listed to reproduce 
the work sheets. A portion of the listing is shown as Table 3. (The 
right-hand column appended to this table will be described in 5.1). 


2.3 The type 6 cards. The type 6 cards carry one-half the sign- 
change sums in fields identical to those used in type 1 cards. These 
sums are calculated from the column totals of Table 4. 


2.31 The sign-change sum. To change the sign of a variable in 
a correlation matrix and adjust the column total accordingly, we need 
only subtract the variable twice from the column sum. Assume for 
the moment that Table 5 is a correlation matrix with communalities 
omitted and the diagonal cells filled with 01.0000, Table 5 is in fact 
a portion of our data with unity values subtracted by means of the 
type 7 cards. 

Variable 18 shows the largest negative sum (—.1022) and is 
therefore selected to have its sign changed first. All coefficients in 
row 18 are subtracted from the totals twice and the remainders are 
the column sums with variable 18 changed to a positive sign. 

But since the tabulator can be made to subtract only once during 
a card cycle, we may secure the same result by dividing the column 
total by 2 and subtracting the variable only once. This is made clear 
by the following equation ; 


sum 2r 
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Actually we need not double the difference on the right side of the 
equation because we need only relative values in order to select the 
proper variables to have their signs changed. 

The column totals in Table 4 contain 80 unities which were added 
according to operation 1.1. Therefore we subtract 80 from each total 
before we divide by 2. No difficulty is experienced when the sums are 
positive, i.e., larger than 80.0000, but when they are negative, i.e, less 
than 80.0000 the calculation may be a bit confusing at first. Since we 
wish the difference to be in complement form, with 2 places carried 
to the left of the decimal, i.e., 100 larger than it actually is, and since 
we are to divide by 2, we merely add 200 to the column total before 
we subtract the 80. These operations are made clear by the follow- 
ing examples: 


79.1130 variable 1, Table 4 80.2081 variable 6, Table 4 
200.0000 
279.1130 
—80.0000 80.0000 
2|199.1139 2| 00.2081 
99.5565 = 1/2 SCE 00.1040 = 1/2 SCE 


These 1/2 sign-change sums are shown on Table 4 just below the print- 
ed column totals and they were punched in 10 type 6 cards according 
to the code shown in Table 1. A new set of type 6 cards was punched 
for each residual table. 


2.33 The preliminary sign-change tabulation. The type 6 cards 
were tabulated to permit us to select the variables with the largest 
negative sums for the first sign-change trial. In order that these 
negative sums will be in direct rather than complement form, the 
tabulator is wired to subtract cards carrying an x in column 7.* Since 
all type 6 cards carry x (7) the values they carry are reversed as they 
enter the counters and negative values are thus expressed in direct 
form. 

Type 8 cards were sorted ahead of the type 6 cards on (6-5) and 
the preliminary sign-change tabulation shown as Table 6 was printed. 
Succeeding sign-change trials will be described as operation 3.- 


2.4 The index cards. Index cards were used to aid in selecting 
the cards for variables which were to be changed in signs. The vari- 
able number was printed by means of a numbering machine upon the 
tab and also punched in (1-3). Index cards cannot be punched except 


* The x-control is sometimes known as the 11 punch. 
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by hand-feeding each card into a numeric key punch. Therefore, the 
numbers must be punched from the reverse side and an 001 was 
punched as 100. An index card for each variable was prepared. 


2.5 The type 7 cards. Type 7 cards were used to subtract the 
one which was added to each r to make 1 + 7 values in the correlation 
tables. (See operation 1.1). These cards were used withomt altera- 
tion in each subsequent sign-change procedure. 


2.6 The type 8 cards. Type 8 cards were used in all tabulations 
to print the variable number as column headings. 


2.7 The type 9 cards. Type 9 cards are punched to carry the 
communalities, i.e., the largest coefficient in each column. These were 
found by sorting the type 1 cards as follows: 


2.71 Sort each detailed field (see codes) for each section on the 
first position to the left of the decimal, i.e., sort first on the column 
indicated under sort 1 below. Keep the 0’s and 1’s separate in sub- 
sequent sorting. Next, sort 1’s on the column indicated under sort 2 
and select cards in the highest sorter pocket. Sort 0’s on the same 
column and select cards in the lowest sorter pocket. 


Sort 1 Sort 2 Sort 3 Sort 4 
Col. (12) Col. (13) Col. (14) Col. (15) 
Col. (18) (19) (20) (21) 
Col. (24) (25) (26) (27) 
Col. (30) (31) (32) (33) | 
Col. (36) (37) (38) (39) 
Col. (42) (43) (44) (45) 
Col. (48) (49) (50) (51) 
Col. (54) (55) (56) (57) 


A plan for determining which of these pockets contains the high- 
est valued r regardless of sign is given below: as 


The sorter pocket farthest removed from the vertical line should con- 
tain the cards carrying the highest r’s. If “0” cards from the first sort 
are in pocket 2 and “1” cards are in pocket 6, only the cards in pocket 
2 need be selected for subsequent sorting. If cards fall in pockets 
joined together in the diagram above both sets must be sorted on sort 


|| 

: | | | | 
| | | 

| | | | 
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3. If there are only 3 or 4 cards in these extreme pockets, they may 
be compared by converting the value on the “0” cards to complements 
and thus selecting the largest r. This value should be recorded on the 
table given below. 


Estimated Communalities 
Fields and Card Columns 


Section 1(12-15) | 2(18-21) | 3(24-27) | 4(30-33) 


| | | 
| | | 
1 | | | | 
| | | | 
3 | | | | 
| | | | 


Section 5 (36-39) | 6(42-45) | 7(48-51) | 8(54-57) 


| | | 

| | | 

| | | 

| | | 

| | | 
ete. | | | | 
The type 9 cards are punched from this table. 


2.8 The Type 10 Cards. Type 10 cards were not punched until 
the sign-change procedure was completed. They contain a correction 
factor which is subtracted from the column totals after sign change 
to remove the unities which remain in the table. If the table contains 
80 variables, 33 of which had their signs changed, 33 X 2 or 66 of the 
unity values would be cancelled out leaving 80 — 66 = 14 unities to 
be taken out by the number 10 cards. This value will be found as the 
total for the check variable (76 in our case, see operation 1.1, and the 
tabulation sheet after sign-change, operation 3.1). 

Cards “Index,” 7, and 8 should be saved for future use; the other 
cards must be punched anew for each residual table. 


3. The sign-change procedure. Prior to sign change, the index 
cards and the type 1 and type 7 cards were count-sorted in the order 
named on (3-1) and stored in a card file. The type 1 cards had al- 
ready been verified (operation 2.21) and the type 7 cards were count- 
ed as they were sorted into the deck. 

The preliminary sign-change trial (operation 2.33) merely per- 
mitted us to select the first variables to have their signs changed. 
These variables having been changed, a new tabulation was obtained 
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which permitted us to select the next group of variables to be changed 
in sign. This procedure was continued until all variables with col- 
umn totals in direct form were changed in sign and all in complement 
form remained unchanged. 

The variable with-the largest column total was always selected 
first, and if there were several of about this same size, these were 
changed simultaneously. We have changed as many as 8 variables at 
one time during the early part of the procedure but toward the end 
it seems best to change only one at a time. 

Following the preliminary sign-change trial variables 38, 48, 59, 
and 74 were circled on the tabulator sheet to indicate that they had 
been selected to have their signs changed because each of their col- 
umn totals was considerably larger than any other. (See Table 6). 

The cards for these variables were pulled from the card file and 
sorted on (6-5). behind the type 8 and type 6 cards taken from the 
tabulator after the tabulation of Table 6. 

The index cards were rejected but the type 8, 6, and 7 cards were 
tabulated, using wiring diagram (see Figure 2). Minor control was 
(5-6) and (80); «(7) was wired to subtract and suppressors re- 
mained as they were in operation 1.1. 

The 3 steps in each sign-change trial are as follows: 


1. One-half the column totals less the unities, carried 
on the type 6 cards enter the counters which were 
wired to subtract x (7). 

2. To the counters were added the 1 + r values from 
type 1 cards. 

3. From the counters were subtracted the unities by 
the type 7 cards carrying x in (7) and (10). 


As a consequence the unities have been removed and the counters con- 
tain 1/2 the sums of r’s from which has been subtracted the 7’s for 
the variables whose signs have been changed. All operations have 
been reversed in order that negative sums may be printed in direct 
form. 

Using the next trial sheet after 38, 48, 59, and 74 have been 
changed, a second set of variables were selected, sorted, and tabulated 
as described above. 

Table 7 was the 7th trial sheet upon which the variable numbers 
of those previously changed ¢shown in italics) were circled. It should 
be noted that variables 21 and 74 have been reversed. This may indi- 
cate that they should not have been changed, but because their nu- 
merical values are smaller than the values of 39, 51, and 61 we did 
not remove them at this time. 


i 
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We used small circles about the variable number to show that 
the variable had been changed in a previous trial. We used a large 
circle about the variable number and its sum to indicate its sign was 
to be changed next; and we used a large X to indicate that the vari- 
able should be removed from the tabulated pack because in changing 
other variables the sign had been reversed. These symbols cannot be 
shown in type so bold face, italics, and underlining were used as in- 


- dicated in the tables. 


Table 8 shows variable 21 to be numerically larger and 74 small- 
er than on Table 7. Our problem was to decide whether to remove 
these variables and return them to the card file or to add new vari- 
ables showing larger totals as do variables 5, 28, 50 and 78. We com- 
pared the true values, i.e, 5 = —.3741, 2i= +.2208, 28 = —.3145, 
50 = —.2428, 74 = +.1233, and 78 = —.4655 and decided to remove 
21, add 5, 28, and 78, and not remove 74 since its total is much smaller 
numerically than any of these. On trial 20 (Table 9) it is seen that 
this choice was justified. 

Table 9 shows sign-change trial 20 for the tenth factor extracted 

‘in our problem. All variables in italics are larger than 0 and all not 
in italics are smaller. Since these values have entered the tabulator 
with «(7) wired to subtract they are expressed as complements. Ac- 
tually the signs of the variables whose column totals were negative 
have been changed to positive values, and the sign-change procedure 
is thereby completed. 


3.1 The sums after sign change. The sums after sign change 
were now obtained directly from the type 1 cards in the following 
-manner: 

The cards which have been tabulated in operation 3 were sorted 
on (4) in order to separate them by type. This portion of the type 1 
cards was gang-punched «(9). These cards which were not tabulated 

were sorted on (4) and all index cards and types 6 and 7 were rejected. 

The table totals needed in operation 4 should be obtained at this 
point, and before continuing with operation 3.1. All type 1 cards were 
tabulated with control on «(9) subtract to obtain 8 field totals which 
were combined into a table total by hand, to equal item B,¢ in Table 
lla. The minor control switch was off. Then the type 9 cards were 
tabulated to obtain 8 field totals which were combined by hand into 
a table total to equal item Cé in Table 11a. 

To continue with operation 3.1, the type 8 cards were sorted 
ahead of the type 1 cards on (6-5) and tabulated with control 2(9) 
subtract. Table 10 shows the column totals after sign-change and the 

table is checked by means of the check variable (76 in our case). All 
variables that have been changed must show larger totals than 76 


: 
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shows and those not changed must be smaller. If more than half the 
variables had been changed the check column would be in complement 
form. The total of this check column is the amount that must be sub- 
tracted from each column total by means of the type 10 cards as ex- 
plained in operation 2.8 and 4. Type 10 cards were then punched to 
carry the total for the check variable. 


4. The Factor Calculation Sheets. The estimated communalities 
(see operation 2.7) were determined and the type 9 cards punched 
while the sign-change trials were being run. 

As soon as operation 3 was completed, the following 4 cards were 
inserted after the last type 1 card in each section: 


1. The type 10 card for the section. 
2. Ablank card. 

3. The type 9 card for the section. 
4. A blank card. 


The pack was tabulated to provide the factor calculation sheet shown 
as Table 11. 

The variables that have been changed were circled to indicate that 
all values for that variable and the factor loading will be negative. 
The rows in Tabie 11 are identified as follows: 


B, row shows the sum of the columns for each variable after ° 
sign-change. 

B. row shows the amount to be subtracted from each column 
total as is obtained from type 10 cards. 

B,; row is the difference between the B, and B, rows ex- 
pressed in direct form. The values in this row must have 
the same sign as that given the variable. Negative num- 
bers can be written in red. 

C row contains the estimated communalities as obtained 
from the type 9 cards. The sign of this item is the same 
as that given the variable with negatives in red. This cell 
is left blank for any variable carried merely to fill out a 
section. 

D row is the sum of the B; and C rows and each total must 
have the sign given the variable. 


The calculations within Table 11 are checked in Table 1la. The 
B,t total was obtained by summing the 8 field totals as described in 
operation 3.1. The B.t total is the amount carried in any one field in 
the B, row times the number of variables in the table. The positive 
and negative values in the B; rows were summed separately. Their 
difference is the B,t value in Table 11a, i.e., the table sum with regard 
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for signs, and it should equal the difference between the B,t and B.t 
totals in Table 1la. The positive and negative values in the C rows 
were summed separately, and their difference is the value of the Ct 
total. Likewise, the + and — values in the D rows are summed sepa- 
rately and their difference is the Dt total of Table 1la, and it should 
equal the sum of the B;¢ total and the Cé total. 

The square root of the sum of the 7 values for the table is ob- 
tained through Table 11b. The sum of the + and — values of the B; 
rows, i.e., the total without regard for signs is entered as B; total in 
Table 11b. The sum of the + and — values of the C rows equals the 
C total and the sum of the + and — values of the D rows equals the D 
total of Table 11b. B; + C = D in this table as a check also. The 
square root of D is found and its reciprocal calculated. This value is 
used as a constant multiplier with each D value in Table 11 to obtain 
the factor loadings for each variable, which are entered to the left 
of the variable number in Table 11. 

The sum of the factor loadings for the table should check except 
for rounding errors with the amount obtained by multiplying the 
sum of the D rows by the reciprocal obtained above. 


5. The residual tables. A blank copy of the work sheets pre- 
pared in operation 1.1 was used for the calculation of the first residual 
table which was known as correlation Table 2. The sign of the load- 
ing of each variable on factor 1 was entered in the middle column; 
plus signs were entered in red pencil so that they could be clearly dis- 
tinguished. (See Table 2.) 

At this time a set of guide strips similar to the one shown to the 
right in Table 3 was prepared to carry the variable numbers and the 
loadings on the previous factor. 

The residuals were then calculated by one worker using the Mon- 
roe calculator, while a second clerk recorded the values on the work 


sheets. 


5.1 Calculating the residual values. The four guide strips were 
clipped to their respective pages of section 1 of Table 1 in such a 
manner that the variable numbers corresponded row by row with 
column 1. 

The factor loading for variable 1 was locked in the Monroe as a 
constant multiplier. Since the signs of the factor loadings which are 
to be multiplied together must be carefully noted, we found it advis- 
able to multiply all plus factor loadings in the column first, then re- 
turn and multiply all minus loadings. The reason for marking the 
sign of the factor loading in each row on the work sheet is now appar- 
ent. The clerk recording must place the values in their proper cells. 
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By operating in this manner we needed to change the negative multi- 
plication lever only once. The lever was set depending upon the sign 
of the factor loading locked in the machine. We wished to multiply 2 
factors and subtract their product from the r of the previous table 
all in one operation, which is made more clear by the following ex- 


amples. 


r placed constant changing residual r taken from 
in lower multiplier multiplier lower dial and copied 
dial locked in placed on on work sheet 
keyboard 

4217 —(-10 -—.10) 4117 

3427 —(-—.10 xX +.10) = 
99.3427 —(4+.10 xX —.10) = 99.3527 
99.4217 —(+.10 xX +.10) = 99.4117 


The rules for operating the multiplication lever when one wishes 
to subtract the product of two factors from a coefficient were written 
on a card kept by the calculator. 


1. Pull the lever forward if the signs of the terms being 
multiplied are different. 

2. Push the lever backward if the signs of the terms being 
multiplied are alike. 


To clear the key board with the 0 bar may change the setting of 
this lever, therefore, we always cleared by using the keys instead. 

The recorder sat at the left of the operator and read the figures 
which appeared on the lower dials at the end of each multiplication. 
By the time the machine operator had set up the next coefficient upon 
the key board the recorder had written the former value upon the 
work sheets. The machine may thus be kept in continuous operation. 
Two operators have been able to calculate the 6400 residuals, punch 
the new set of cards and verify the calculations in 48 hours. The 
operators alternated positions frequently. 


5.2 Verifying the calculations and punching. Generally it seems 
best to calculate the residual correlations column by column, and thus 
each is calculated twice and the table can be verified by reading col- 
umns against rows. As our operators became familiar with the pro- 
cedure, we calculated only that half of the residual table below the 
diagonal. The values thus calculated were entered in the correspond- 
ing cells above the diagonal and sight-verified. The type 1 cards were 
punched from these work sheets after 01.0000 had been substituted 
for the value in each diagonal cell. These cards were sorted as described 
in operation 2.21 and tabulated to obtain section and unit totals as de- 
scribed in operation 2.22. 
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While the type 1 cards were being punched, the section totals for 
the next residual table were calculated in order that the cards might 
be verified as soon as the section totals were tabulated. The verifica- 
tion proceeded as follows: 

The column total for variable 1 obtained from the original 1 plus 
r matrix was placed in the lower dial of the Monroe calculator. The 
sum of the loadings for factor 1 was locked in the Monroe as a con- 
stant multiplier. The factor loading for variable 1 was placed on the 
keyboard. The minus multiplication lever was set according to the 
rules cited in operation 5.1. The multiplication was then completed. 
However, since the communalities have been omitted from the type 1 
cards and hence from the total, the square of the factor loading for 
variable 1 must be added. This is done by pulling down both the con- 
stant multiplier lever and the minus multiplication lever and squaring 
the factor loading on the keyboard. This figure must equal the col- 
umn total of variable 1 as tabulated in operation 2.22 for the first 
residual table. To verify the total for column 2 the factor loading for 
variable 2 is used in the same way. This operation is similar to that 
of calculating the residual table. It differs in that column totals are 
used in place of individual coefficients and individual factor loadings, 
and in that an adjustment is made for having omitted the communal- 
ities in the type 1 cards. Each residual table is verified in this manner 
from the totals and factor loading of the preceding table. 

If an error is found in a column total it can be isolated within a 
unit by applying this verification procedure to unit totals. Adjust- 
ment for communalities need not be made except in the unit contain- 
ing the diagonal cell. 

The factor loading tabulation sheet shown as Table 12 assists one 
in making the above calculations since it contains all the multipliers 


needed. 


Appendix—Summary of Instructions for Each Tabulation 


1.1 The work sheets. 

Sort 10 type 8 cards for section 1 ahead of type 0 cards on 
(79-80). Tabulate using wiring diagram (see Figure 2). Add 
counter-list wires. Pull up short block-out levers 14-15, long 
hammer-loek levers for all type bars used and zero suppressers 
for 11, 16, 23, 29, 36, 48, 6, 18, 20, 27, 28. Set carriage to double- 
space and to print 9 4/6 inches on a 11-inch sheet, 4-copy paper. 
Sort out the type 8 cards on (80) and sort in the 10 type 8 cards 
for section 2. Tabulate as before. Repeat these operations for 
all 10 sections. Save cards for future use. 


2.21 Tabulating column totals, 
Use tabulator setting as in 1.1. Count-sort type 1 cards on (3-1). 


: 
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Count-sort type 8 cards ahead of type 1 cards on (6-5). This is 
for verification. Tabulate with minor control on (5-6) and (80). 
One copy. Tabulate units total by changing control wires to 
(78-79) and (80). 


2.22 Listing the correlation matrix. 
Counter-list cards used in 2.21 with minor control on (78-79) 
and (80). The long hammer-lock levers are lowered for this op- 
eration. One copy. 


2.33 Preliminary sign-change tabulation. 
Sort type 6 cards behind type 8 cards on (6-5). Tabulate with 
minor control on (5-6) and (80) and x subtract control on (7). 
Pull up long hammer-lock levers again. One copy. 


3.0 Sign-change trial sheets. 
Select variables from tabulation 2.33 which show largest totals. 
Pull index cards, type 1 and type 7 cards from file box for these 
variables. Sort 8, 6, index, 1 and 7 cards in order given on (6-5). 
Tabulate with tabulator set as in 2.33. 


3.1 Sum after sign-change. 
Sort cards that have been tabulated in final trial of 3.0 on (4) 
in order to separate cards by type. Gang-punch 2(9) in this 
portion of the type 1 cards. Sort cards remaining in card file at 
end of operation 3.0 on (4). Combine the type 1 cards here with 
those that have been punched x(9) and tabulate without minor 
controls (but change x subtract to (9)), in order to obtain 8 to- 
tals which may be summed into a table total. This is the B, total 
of Table 10b. Tabulate the type 9 cards without minor control 
in order to obtain C total of Table 10b. 
Now sort type 8 cards ahead of type 1 cards on (6-5) and tabu- ~~ 
late to sum after sign-change. This step may be omitted as it is 
the same as the B, row on the factor calculation sheets. 


4.0 Factor calculation sheets. 
We generally tabulate the type 1 and type 9 card totals as de- 
scribed above on the top of the calculation sheet. 
The cards are arranged as follows for each section: 


The type 8 cards 

The type 1 cards 

The type 10 card for the section 
A blank card 

The type 9 card for the section 
f. A blank card 


These are tabulated with minor control on (5-6) and (80) and 
x(9) subtract. 


i 
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FIGURE 1* 
Work Sheets, for Entries of Original Correlation Matrix and Each Succeed- 


ing Residual Matrix 


* Figure 1, Tables 3, 4 and 6-11 were set up on the tabulator. 


| 
21 : 
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TABLE 1 
Codes for Cards Used 
Card type 
General layout 
Index 0 1 6 7 8 9 10 

Card form Index 5080 5080 5080 5080 5080 5080 5080 

Card color Manila Manila Manila Salmon Green Yellow Salmon Manila 

No. of cards 80 1/ 80 800 10 800 10 10 10 
Col. 
No. 
(1-3) Variable No. use use use blank use blank blank blank 
(4) Card No. blank 0 1 6 7 8 9 blank 
(5-6) Section No. blank use use use use 2/ use 
(7) blank “J” 4/ blank blank blank 
(8) blank x5! blank “ 
(9) x 6/ blank “ “ “cc 
(10) “ “ blank “ 

8/ 9/ 

Detail fields 
(11-16) Field1 blank blank use blank (15-16) 10/use 11/ use 12/ 
(23-28) 3 “ “ 1 “ “6 (27-28) “ 
(29-34) 4 “ “ 1 + Tr “ “ (33-34) iii “ 
(35-40 5 “ “6 1 +r “ “ (39-40) “ “ 
(41-46) 6 “ “ 1 “ (45-46) “ “ 
(47-52) “ 1 + “ “ (51-52) “ “ 
(76-77) Table No. “blank _use13/ use blank blank use use 
(78-79) Unit No. * use usel4/ blank “ use biank blank 
(80) omit blank 1 15/ x (80) 16/ 


1/ This problem had 79 variables so 1 blank was entered to fill out the table. 
_2! x (6) also to split control. 3/4/ x (7) subtract control. 5/ x (8) subtract. 


6/ x (9) subtract only those variables that have had signs changed. 


4/ x (10) subtract control for 1 in (7). 

8/ Columns and decimal point identical with card 1. 

9/ Use only 5th and 6th column of each field. 

10/ Punch 0 in first column of each field. 

11/ Decimal preceded by 00. 12/ Punch all 6 digits. 

13/ Original table = No. 1, each residual table 1 unit higher. 

14/ Number each 8 X 8 unit in each section. 15/ 1 for minor control. 
16/ x (80) to split control. 


; 
4 
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Work Sheets Showing Residual Correlations for a Part of the Table for Fac- 
tor 10. (The original entries in this table were made by hand on a work sheet but 
are set in type here. See operation 1.1) 


1 
49 | 010334 
50 | 010071 
51 | 010739. 
52 | 010040 
58 009010 
54 | 009325 
55 010353 
56 008933 

1 
57 8515 
58 9410 
59 9987 
60 9215 
61 9887 
62 9878 
63 11258 
64 8120 

1 
65 10169 
66 9863 
67 8145 
68 9451 
69 11095 
70 10935 
71 10203 
72 9945 


011300 + 
013047 + 


4 


8806— 
8851+ 
5983+ 
9941+ 
5854+ 

10914+ 

11656— 
9751— 


4 


10602+ 
9445+ 
10601— 
-11249— 
10836 + 
12080+ 
-11877— 
11438+. 


011154 
009192 
008777 
010140 
009175 
010911 
010176 
010384 


| | | | 
| 009456 | 009279 | 010604 — | 009856 | 009908 | 010682 
009810 | 008254 | 009608-+ | 010568 010313 | | 009675 
| 010290 | 009995 | 009979—: | 010008 | 010080 | 007804 
| 010772 | 010529 | 011825— | 010804 | 010604 | | 010011 : 
| 009086 | 009761 | 009572— | 010184 | 009368 | | 008347 7 
| 010770 | 008196 | 009150— | 010181 | 011042 | | 010417 i 
| 010048 | 010169 | po | 010480 | 010278 | | 010927 
| 010770 | 011148 | | 010658 | 009940 | | 010673 
| | | 
| | | | | j 
| 9232} g12e | | 10271} 9760} 9409 | 10562 
| 11518 | 9098 | | 9576 | 10516 | 9458 | 9788 
| 11421 | 10719 | | 9284 | 10584 | 10960 | 10855 
| 8936 | 10119 | | 10891 | 10824 | 10252 | 10621 ; 
| 9187 | 9882 | | 11882 | 9791 | 11281 | 11967 
| 10854} 8187 | | 9684} 9809] 9677 | 10712 
| 10770 | 9768 | 9824] 10178} 8470 
| 7261 | 10088 | 9064 | 10851 |} 10131 | 10984 
| | | | 
| | 
| 8999 | 8270 | | 11245 | 10290 8794 | 9580 
| 10696 | 11128 | | 9007 | 9595 11192 | 8523 
| . 9852 | 10124 | | 10093 | 10884 | 10965 | 11497 
| 9667 | 9458 | | 10721 | 10309 | 10910 | 10629 : 
| 8567 | 11010 | | 11228 | 10078 | 10412 | 9594 
7810 | 10124 | 9890 | 9569 | 10024 10847 ‘ 
9666 | 10267 | | 9588 | 10228 | 9599 9172 
| 9482 | 9480 | | 9187} 10850 | 10251 | 10475 
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TABLE 3 
The Tabulation of Work Sheets Shown in Table 2 with Unit Totals as the 
8th, 16th, and 24th Rows. A strip clipped to this table was used in the calcula- 
tion of the residual value. (This table was listed on the tabulator but is shown 
here set in type. The figures to the right were entered by hand on a strip of tabu- 
. lator paper so that the entries matched exactly row by row with the tabulated 


sheet.) 
Variable Leading 

No. Factor 

9 
49 01 10334 9456 9279 10604 9856 9908 49 —.08 ; 

10071 9810 8254 9608 10568 103813 50 +.18 

10789 10290 9995 9979 10008 10080 51 —.15 
10040 10772 10529 11325 10804 10604 52 — 32 

9010 9086 9761 9572 10134 9868 58 

9325 10770 8196 9150 10131 11042 5A —.05 

10853 10048 10169 11300 10430 10273 55 4.25 
8983 10770 11148 13047 10658 9940 56 +.24 3 

49 01 78805 81002 77331 84585 82589 81528 

57 01 8515 9232 9122 8806 10271 9760 57 —.06 3 

9410 11518 9098 8851 9576 10516 58 +.10 

9987 11421 10719 5983 9234 10584 59 +.20 
9215 8936 10119 9941 10891 10824 60 +15 4 

9887 9137 9832 5854 1189382 9791 61 +.21 

: 9878 10354 8137 10914 9684 9809 62 +.08 
11258 10770 9763 -11656 9824 10173 63 —.18 : 

8120 7261 10088 9751 9064 10351 64 +.12 

57 01 76220 78629 76828 71756 79826 80808 

65 01 10169 8999 8270 10602 11245 10290 65 +.08 8 

9863 10696 11128 9445 9007 9595 66 +.26 

8145 9852 10124 10601 10098 10884 67 —34 
9451 9667 9458 11249 10721 10309 68 —36 3 

11095 8567 11010 10836 11228 10073 69 +.17 

10935 7810 10124 12080 9890 9569 70 +.22 
10203 9666 10267 11887 9588 10228 71 14 3 

9945 9482 9480 11488 9187 10350 72 4.21 


— 
. 65 07 79806 74739 79811 88138 80959 81298 = 


TABLE 4 

The Column Totals for Each Variable Tabulated from the Cards. [Column 
totals shown in ordinary type were printed on the tabulator. Those in bold face 
(top row) were obtained in operation 5.2 and were entered by hand. Those in 
italics (lower row) were obtained in operation 2.31 and entered on this sheet by 
hand in red.] : 
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iJ 2 3 4 5 6 uf 8 
791130| 797143 788973 797278 796099 802081 806854 798551 
01 791130) 797143 | 788973 | 797278 | 796099 | 802081 | 806854 | 798551 
995565| 998571 994486 998634 998049 001040 003427 899275 

2 9 10 11 12 13 14 15 16 
798256; 810913 799208 793541 805830 796145 789734 799457 
02 978256; 810913 793541 805830 796145 789734 799457 
999128| 005456 999604 996770 002915 998072 994867 999728 

3 17 18 19 20 21 22 23 24 
792270| 801795 793850 812386 788522 790187 799368 796881 
03 792270} 801795 793850 812386 788522 790187 799368 796881 
996135| 000897 996925 006198 994261 995093 999684 998440 

4 25 26 27 28 29 30 31 32 
799882 | 798717 799936 801287 801781 798286 809906 805479 
04 799882} 798717 799936 801287 801781 798286 809906 805479 
999941| 999358 | 999968 000643 000890 999143 004953 002739 

5 33 34 35 380 387 38 39 40 
800409 | 801460 807703 794729 805190 780331 790696 795173 
05 800409) 801460 807703 794729 805190 780331 790696 795173 
000204; 000730 003851 997364 002595 990165 995848 997588 

6 41 42 43 44 45 46 47 48 
814152; 800176 814646 808492 793809 806959 818154 786270 
06 814152| 800176 814646 793809 806959 818154 786270 
007076| 000088 007323 004246 996904 008479 009077 993135 

7 49 50 51 52 53 54 54 56 
794104; 801744 790932 796805 795528 806263 810106 802094 
07 794104) 801744 790932 796805 795528 806263 810106 802094 
997052| 000872 995466 998402 997764 003131 005053 001047 

8 57 58 59 60 61 62 63 64 
803008 | 790526 782428 792551 780322 791895 800029 799257 
08 803 790526 782428 792551 780322 791895 800029 799257 
001504| 995263 991214 996275 990161 995947 000014 999628 

9 65 67 68 69 70 71 72 
799460| 808599 811634 816528 792378 812788 794609 788136 
09 799460} 808599 811634 816528 792378 812788 794609 788136 
999730| 004299 005817 008264 996189 006394 997304 994068 

1 8 74 75 76 77 78 79 80 
809367| 785111 791545 788318 795486 789841 803859 
10 809867) 785111 791545 800000 788318 795486 789841 803859 
004683| 992555 995772 994159 997748 994920 001929 


3 
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TABLE 5 
A Portion of Table 3 with the Unity Values Subtracted 
Variables 15 16 17 18 19 20 21 
15 0000 09384 —.1203 1719 0858.  .0120 —.0099 
16 0934 .0000 —.0174 —.1951 1322 0646 0057 
17 —.1203 —.0174 0000 .0460 —.0019 .0463 —.0481 
18 1719 —.1951 .0460 .0000 —.0982 —.0825 .0557 
19 0358 1322 —.0019 —.0982 —.0525 —.0486 
20 0120 .0646 0463 —.0825 —.0525 .0000 .1595 
21 —.0089 —.0481 0557 —.0486 1595 
Sum -1829 0834 —.0954 —.1022  —.0332 .1474 .1143 
Calculations when 2 7,, is subtracted from the column totals 
Sum .1829 0834 —.0954 —.1022 —.0330 .1474 
2115 3438 —.3902 .0920 0000 —.1964 —.1650 1114 
Diff. —.1609 —.3068 —.0034 —.1022 +99.8368 —.0176 +.0029 
Calculations when 7,, is subtracted from 1/2 the column totals 
1/2 Sum 0914 0147 —.0477 0511 —.0166 .0571 
1719 —.1951 —.1022 —.0982 —.0825  .0557 
Diff. —.0805 —.1534 —.0017 —.0511 +9.9174 —.0088  .0014 
X 2forcheck —.1610 —.3068 —.0034 —.1022 +99.8348 —.0176 -+.0028 


TABLE 6 


Preliminary Sign-Change Trial Using Only Type 8 and Type 6 Cards. (See 
operation 2.33. These entries were tabulated and the figures in bold face were 


circled to show the variable was changed in sign.) 


1 | 2 3 4 5 6 7 8 
1 4435 1429 5514 1366 1951 998960 996573 725 
2 9 10 11 12 13 14 15 16 
2 872 994544 396 8230 997085 1928 5133 272 
3 17 18 19 20 21 22 23 24 
3 3865 999103 38075 993807 5739 4907 316 1560 
4 25 26 27 28 29 30 31 32 
d 59 642 32 999357 999110 857 995047 997261 
5 33 34 35 36 37 38 39 40 
5 999796 999270 996149 2636 997405 9835 4652 2414 
6 41 42 43 44 45 46 47 48 
6 992924 999912 992677 995754 8096 996521 990923 6865 
7 49 50 51 52 53 54 55 56 
yf 2948 999128 4534 1598 2236 996869 994947 998953 
8 57 58 59 60 61 62 63 64 
8 998496 AT37 8786 3725 9839 4053 999986 372 
9 65 66 67 68 69 70 71 72 
9 270 995701 994183 991736 3811 993606 2696 5932 
10 73 74 75 76 77 78 79 80 
10 995317 7445 4228 5841' 2257 5080 998071 
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TABLE 7 
Trial Sheet upon Which Variable Numbers of Those Previously Changed Have 
Been Circled. Variables 39, 51, and 61 are to be Changed Next. (These entries 
were all tabulated, and the variable numbers in italics were circled to show that 
they had been changed, while those in bold face were circled to show that they 
were to be changed in the next operation.) 


4 
988267 
12 


1 
1 
2 
2 
3 
3 
4 
4 
5 
5 
6 
6 
if 
7 
8 
8 
9 
9 
0 


1 
10 


119 
1 2 3 5 6 7 8 - 
997757 9973875 9751 3741 997306 5426 4385 
9 10 11 13 14 15 16 
: 994942 990982 993407 4491 995792 994180 7015 995071 gg 
: 17 18 19 20 a1 22 23 24 
998371 1884 999166 996807 997792 2137 995149 55 
25 26 27 28 29 30 31 82 & 
879 1534 993698 3145 991230 6716 991640 994745 : 
33 34 35 36 37 38 39 40 
999686 996661 991319 4891 997525 10540 3949 985676 
: 41 42 43 44 45 46 47 48 .: 
988286 992094 994226 997414 993129 991348 987414 4976 
49 50 51 52 53 54 55 56 & 
998760 2428 3932 995396 996753 994633 996254 369 a 
: 57 58 59 60 61 62 68 64 & 
997770 992685 5856 5568 4074 996332 994793 1867 
65 66 67 68 69 70 71 12 a 
998983 994867 995775 990254 5641 996794 998568 1888 
73 7% 75 76 77 78 79 80 , 
999244 998767 1975. 8552 2824 7157 992599 


_PSYCHOMETRIKA 


TABLE 8 

The 8th Trial Sheet; Variables 5, 28, and 78, Are to Have Their Signs 
Changed Next (These entries were tabulated originally. Then the variable num- 
bers in italics were circled to show that they had been changed previously. Those 
in bold face, that is, 5, 28, and 78, were enclosed in a large circle to show 
that they were to be changed next, and the figures underlined were crossed out 
with a large X to indicate that the cards for 21 were removed because changes 
in other variables had reversed its sign. 


a a 2 8 4 5 6 7 8 
1 995197 994346 8109 987039 5293 997270 5323 4502 
2 9 10 11 12 13 14 15 16 
2 992861 989001 992523 6003 997849 994949 9122 994894 
3 17 18 19 20 21 22 23 24 
3 994264 2342 341 992514 994227 1382 995922 613 
4 25 26 27 28 29 30 31 82 
os 1892 276 998028 4675 991381 5644 992665 994828 
5 33 34 35 36 37 38 89 40 
5 999422 995162 991612 5699 2485 10846 4135 984991 
6 41 42 43 dd 45 46 AT 48 
6 984999 994390 994338 997069 992649 992936 987613 72A2 
7 49 50 51 52 58 54 55 56 
7 991891 2467 5532 993631 998251 994825 999521 998437 
8 57 58 59 60 61 62 63 64 
8 999116 991938 5353 5810 996754 994371 406 
pie 65 66 67 68 69 70 71 72 
9 998200 993838 994598 992210 7795 995334 75 997215 
0 73 74 75 76 7 78 79 80 
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TABLE 9 
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The 20th Sign-Change Trial. All Variables Shown in Italics and Whose 
Signs Should Have Been Changed Are Now Changed. 


1 1 2 3 4 5 6 7 8 
1 991932 995006 651 985047 3891 994651 6219 4277 
2 9 10 11 12 18 14 15 16 
2 988487 996601 986603 11887 406 993076 11130 998667 
3 17 18 19 20 21 22 23 24 
3 994914 1900 4147 998881 989590 3501 998010 6207 
4 25 26 27 28 29 30 31 32 
4 9798 992322 995860 9948 993306 5972 994394 994380 
5 33 34 35 36 87 38 39 40 
5 994327 994842 992252 2274 6680 10477 4175 988712 
6 41 42 43 Lh 45 46 | AT 48 
6 986153 996977 998431 3323 988256 997944 989077 2522 
1 49 50 51 52 58 54 55 56 
7 986623 6979 8124 992999 992774 1546 996699 992751 
8 BT 58 59 60 61 62-68 64 
. 998448 991792 997578 11049 6870 992106 999695 3702 
9 65 66 67 68 69 7 71 72 
9 6667 990699 992490 992685 4077 994560 997643 997219 
10 73 7h 75 76 77 78 79 80 
10 995097 3985 14351 2547 9308 10397 983533 
TABLE 10 


The Column Sums after the Sign-Change Procedure Has Been Completed. 


(The variables shown in italics have had their signs changed.) 


1 


1 
156136 
9 
163026 


2 
149989 


74 
132131 


169916 


160820 
29 
158389 


6 
150699 
14 


. 3 5 7 8 a 
188699 182219 127562 131447 
2 10 11 12 18 15 16 2 
146799 166794 116227 189188 153849 117740 142667 
3 17 18 19 20 21 22 23 24 . 
1 08 150172 136201 181706 152238 182999 148980 127587 Re 
4 25 26 28 80 81 82 
1 04 120404 155357 148280 120105 128056 151212 151241 
5 33 84 85 86 $7 38 89 40 gg 
1 065 151347 150816 155497 185453 126640 119047 131650 162577 ee 
6 41 42 43 44 45 46 47 48 & 
1 06 167694 146046 148188 1838354 163489 144113 161846 184956 3 
is 49 50 51 52 58 54 55 56 ' 
1 O07 166754 126042 128752 154003 154452 186909 146602 154498 & 
8 57 58 59 60 61 62 638 64 = 
1 08 148104 156416 144844 117908 126260 155789 140611 132597 i 
9 65 66 67 68 69 70 i ; 
1 09 126666 158603 155020 154680 181846 150880 144715 145562 _ 
10 73 73 76 77 78 79 80 
1 10 149807 111299 140000 134906 121884 119207 172985 os 
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TABLE 11 


GT) ON- ot  j%$. tot 6 tht 
Q eel  Qsool-  @vuc- Zobel 


The Factor-Loading Calculation Sheets (See operation 4) 


| 
| 
| 
| 
; | 
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TABLE lila TABLE 11b 
The Verification Table for Sign-Change Procedure Table Used to Calculate 1/r, 
Table total B, rows =B,t 1142.3896 Table totals without regards 
Correction factor, total B, rows =B,t 1120.0000 for signs 
Table total B, rows =B,t 22.8896 
Communalities total = Ct 4.3772 | Table total B, rows 30.5106 
Communalities total 6.3000 
Table total D rows == Dt 26.7668 
: Table total D rows 36.8106 
Sum of factor loadings 4.39 Vr, — V36.8106—= 6.067174 
Calculated factor loading total D, X 1/Vr,= 4.41 Ee xe 16482 
TABLE 12 
Factor-Loading Tabulation Sheet 
Unrotated factor loadings for each variable 
Variable 1 2 3 4 5 6 7 8 9 10 
1 —03 +.42 +.10 +.07 4.27 —10 4.31 
2 +384 +.22 +.27 +.16 +.19 —29 +.10 +.10 —15 +.20 
3 +.21 —12 —18 +.17 +.11 —13 —10 +.11 —.08 —.05 
4 +.23 —.27 —42 +.19 —40 —46 —30 —21 —25 +.57 
5 —03 +.06 +.24 —.02 +.13 —22 -+.05 +.03 —20 —.16 
6 —13 —03 +.15 —29 +.08 +.05 +.17 +.01 —07 +.20 
7 —10 —26 +19 +.17 +.08 —32 +.10 +.22 —.21 —.28 
8 +.09 —28 4.12 +.15 —26 +.14 —16 —.22 
Section Total +.78 —26 +.77 +.60 +.41 —136 +.37 |+.60 —1.22 +.57 
9 +.42 +.14 +42 —85 —31 4.25 —22 +.29 +.04 4.48 
10 +.59 +.17 —41 —09 +.25 —14 +.25 —.22 —.16 +.18 
11 —48 +.07 —18 +.11 —06 4.13 —17 +.13 
12 —.26 —26 —26 +.381 +.387 —13 +.11 —10 +.04 —44 
13 —.28 —383 —09 +.14 +.27 —.06 +.09 +.07 —16 —.08 
14 —05 —25 —15 —21 +.30 —18 +.18 —.10 +.16 +.26 
15 +60 +.04 —07 +.27 +.26 —14 —18 +.27 —13 —40 
16 —48 —06 +.18 .+.28 —11 +.17 +.14 +.08 +.14 +.09 
Section Total +102 —48 —61 +41 +.97 —16 +438 +.12 +.06 +.52 


« 
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TABLE 12 (Continued) 


Factor-Loading Tabulation Sheet 
Unrotated factor loadings for each variable 


Variable 


Section Total 
Variable 

33 

34 

85 


47 
48 
Section Total 


1 
+.29 
—.36 
+.73 
—.53 
—.27 
+.44 
+.58 
—.73 
+.15 
—.60 
—.31 
—.27 
—.39 
+.38 
—.28 
—.11 
+.05 

—1.53 


2 
—.12 
+.08 
+.26 


3 


4 


5 


6 7 
+.20 —.11 
—18 +.15 
—16 +.28 
—15 —.12 
—.02 —.13 
+.15 —.16 
—19 —.07 
+.15 —.27 
—.20 —.43 
+.21 —31 
—.23 —.20 
—14 +.27 
—.25 —.11 
+.26 —.30 
+.17 —.17 
—18 —.08 
+.21 +.14 
+.05 —.76 

6 7 
+.06 +.24 
—.09 +.17 
+.18 +.04 
+.138 +.07 
—.33 —.13 
+.31 +.18 
+.10 +.12 
—07 +.07 
+.29 +.76 
—12 +.24 
—12 +.39 
+.27 —.25 
—.20 —.31 
+.25 +.36 
+.03 +.26 
—11 —.10 
—.07 
—07 +.52 


la, m 


17 +21 =—21 +.14 +.09 +.21 
18 —36 —38 —..0T +29 —13 —.10 
19 89 +88 +12 —16 
20 +.03 +.08 —04 —.13 +12 —06 +26 
21 —05 +.10 —.14 —12 —19 +.87 
22 +01 —11 —17 +.17 —17 +31 —2 
23 +20 +.10 +.18 +.84 +28 
24 —.28 +.36 +.13 +.04 
Section Total +.34 —16 —.60 +.48 +.16 —22 +.26 
25 +07° —21 +16 —38 +06 +11 
26 +32 +08 —14 +82 
27 +38 +34 i+.17 +.23 i-+.18 
28 +11 +08 +09 —.12 +18 —.05 —387 
29 +H —15 +.29 
30 —06 —14 +04 +26 +.27 
31 +32 +.16 +.17 +28 
32 —19 +0 —12 —19 +18 —18 +21 P 
+67 —54 —35 +82 +92 +.26 +.25 
: 1 2 3 4 5 8 9 10 
+.44 +.08 +.05 +.04 —11 —27 -+.28 
; +36 — +23 ~~ 19 +23 —.09 +.23 
+3 —31 +38 +237 —2 +24 +30 
36 +.05 |+.18 —15 +.16 —.16 — 12 
, 37 +.57 +.24 +.11 +.11 +.19 +.04 +.19 —.27 
38 —A8 +28 +.14 +29 +.12 +09 —22 
39 +.57 +.29 +.22 —.09 +.06 —17 —11 —17 P 
40 —16 —18 +.08 +.09 +.06 +18 —05 +.45 
Section Total 41.71 +.08 +.27 +1.01 +.28 —15 —41 +22 
41 —24 +.22 +.16 +.05 +.12 +.28 +.50 
42° +48 +.14 +.22 +.14 
43 —Al —26 +22 —09 +.18 —19 —22 +200 
44 +48 +.84 +.10 +.06 '+.05 —82 —.16 
45 —13 +22 +12 —24 +.44 
46 —19 +18 —11 +1 +.08 +.27 +.11 
—15 —04 +09 —11 +.15 —07 +.11 +.40 
+50 —22 +35 4.88 4.11 
+90 +.06 +26 —42 +.67 —.57 +.77 +1.40 
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TABLE 12 (Continued) 


Factor-Loading Tabulation Sheet 
Unrotated factor loadings for each variable 


Variable 1 2 3 4 5 6 7 8 9 10 
49 +.45 +.138 +.39 —35 —12 +.08 —36 +.31 —.03 4.48 
50 +.10 +.19 —26 +.238 +.18 +.03 +.10 +.29 +.18 —.26 
51 +44 —11 +.20 —12 —19 +.12 —10 —.22 —15 —.32 
52 —49 —14 —11 +.15 —11 +.28 —09 +.09 —32 +4+.28 
58 —.06 —14 —383 +.20 —18 —29 +.23 —16 +.34 
54 +.75 +.28 +.15 —14 —10 +.05 +32 —.13 —.05 —.09 
55 —388 +.12 —23 +.09 —16 +.06 —10 +.11 +.25 +.16 
56 —16 +32 +.08 +.06 +.15 +.18 +.15 —14 +.24 +.29 

Section Total +126 +.68 —41 —.13 —38 +.53 —09 +.88 
57 +65 —.08 +.05 —29 +.17 —10 —22 —.04 —.06 +.09 
58 +.23 +.11 +.18 +22 —22 +.28 —12 +.10 +.32 
59 —12 —09 +.387 —16 +.08 +.10 +.16 +.22 +.20 +.15 
60 +.54 +.19 +.26 —16 —14 +.05 +.16 —16 +.15 —43 
61 —.57 +.19 +.22 +.88 +.18 +.19 —20 +.30 +.21 —.31 
62 —02 —23 +338 +.20 —23 —14 +.06 —20 +.08 +.30 
63 +.23 +.42 4+.22 —.09 +.18 +.12 +.09 +.82 —.18 +.05 
64 +.41 —13 —10 —12 —15 4+.28 +.10 +.27 +.12 —.17 

Section Total +.38 +1538 —02 —18 +.88 +.88 +.59 +.62 .00 
65 +.08 +.24 +.12 +.083 +.11 +.20 —24 +.26 +.08 —.27 
66 +.36 —20 +.05 +.07 —08 +.15 +.08 —.08 +.26 +.34 
67 —08 +.25 —15 +.26 +.21 +30 —17 —21 —34 +4+.32 
68 —12 +.16 --23 +.18 +.22 +.29 +.18 —21 —35 +4+.33 
69 —21 +.29 +.16 —10 +.14 —24 +.06 —.03 +.17 —.16 
10 +.18 —24 —07 +.10 +22 +.17 +.10 +.11 +4+.22 +4+.23 
71 —45 +.26 +.80 —20 —.02 +.20 +.18 —14 +4+.11 
72 —21 +.320 +.12 —11 +.17 —14 +.11 —10 +.21 +4+.18 

Section Total —50 +.92 +.26 +.73 +.79 +.71 +.22 —138 +.11 +1.03 
73 +.22 +.07 +.15 +.16 +.11 +.08 +.09 +.09 +.04 +.20 
74 —50 —06 —10 4+.26 +.14 —10 —07 +.10 +.33 —.17 
15 —29 —47 +.04 —11 —.07 +.07 +.05 —17 —12 —.54 
76 00 00 00 00 00 00 00 00 00 00 
77 +.32 +.05 —11 +.21 +.15 +.12 +.05 +.10 —.11 
78 +.29 —20 —24 +.22 —19 +.18 ++.06 +.13 —18 —36 
19 +.383 —22 4+.17 +.24 +.10 +.19 +.11 —11 —.38 
80 +.22 +.07 +.14 +.28 +.84 +.28 —.09 +.10 +10 +.62 


Section Total —76 +.04 +126 +.55 +27 +14 
Column Total 563 158 190 221 381 151 140 226 02 4.9 
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A NOTE ON RELIABILITY 


HYMAN B. KAITZ 


SOCIAL SECURITY BOARD 
(on military leave.) 


A formula for internal consistency reliability is developed with- 
in the framework of the analysis of variance. The test items are 
assumed to be homogeneous, but may have any weights. Data needed 
for computation are the student test scores, and the total number of 
items answered so as to have the same weight. It is shown that this 
formula reduces to the Kuder-Richardson (21) for item weights of 
one and zero. Some empirical validation is offered. 


When both the distribution of total scores and the item count of 
a test are available, it is convenient to estimate the reliability by the 
use of Kuder-Richardson formula (20).* When only the distribution 
of total scores is available, a simple and adequate approximation to 
the reliability coefficient can be made with Kuder-Richardson formula 
(21). As these formulas were originally stated, they applied only to 
the case of a test consisting of items weighted one (right) or zero 
(wrong, or omit). The concept of K-R (20) was put on a broader 
foundation when it was shown that this formula could be stated in 
terms of analysis of variance components.} In addition, its derivation 
was given rigor by the method of estimation of error variance. Any 
weighting of items whatever could be used in the Hoyt formula. For 
specific methods of weighting test items, a number of variations of 
K-R (20) were offered by another author.{ There is, however, no 
general formula similar to K-R (21) available for a system of item 
weighting other than one or zero. Such a formula, the conditions for 
its use, and some empirical results are reported below. 


Development 


Hoyt§ expressed in equation form the assumption that the score 
of an individual is the sum of four independent factors: 


ye, (1) 


* Kuder, G. F. and Richardson, M. W. The theory of the estimation of test 
reliability. Psychometrika, 1987, 2, 151-160. 

+ Hoyt, C. J. Test reliability estimated by the analysis of variance. Psycho- 
metrika, 1941, 6, 153-160. 

t+ Dressel, P. L. Some remarks on the Kuder-Richardson reliability coeffici- 
ent. Psychometrika, 1940, 5, 305-310. 
§ Op. cit. 
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where 


Xi; = observed score of individual s on item 7; 

A = factor common to all individuals and all items; 

t; =factor associated with item 7; 

p, factor associated with individual s; and 

Yis error factor, normally distributed, with zero mean. 


Using this assumption, he derived the best linear estimate of er- 
ror variance in the sense of least squares, and the best estimate of 
the true variance among individuals as the difference between the 
variance estimate “among individuals” and the error variance esti- 
mate. The reliability is then written as the ratio of the true variance 
to the observed variance, thus: 


V; 


, (2) 


where 


V, = variance estimate among individuals and 
e = variance estimate of error (interaction). 


If now, we make the assumption, far less exacting than the one 
above, that an individual’s score is the sum of three factors, we may 
write 

Xi,=At+D + (3) 
where the factors have the same meaning as before, but we are assum- 
ing that all items have the same factor, which will therefore be in- 
cluded in A, the common factor. By following the same procedure 
as Hoyt, and minimizing the sum of squares, > S yi,?; with respect 
to A and p,, we shall find that the best estimate of error variance is 
what is commonly known as the variance “within individuals.” Con- 
sidering the sums of squares involved, we may separate the total into 
two components: the sum of squares due to differences among indi- 
viduals (SS,), and the sum of squares within individuals (SS,,.). 
This is 


SS;=SS, + SS... (4) 
The best estimate of error variance is 
SS. 

(5), 


where n, is the number of individuals and n; is the number of items. 
Our reliability formula may be expressed as 
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Ve 


Vs 


son formulas by a simple algebraic change. This is 


(nin, — 1) L Ns Ni nyn,—1 
T tt = ? 
os” Ns (nj | 1) 


X, is the mean of the same scores. 


je 


K 
(n,— 1)n; Ni w?? 
nn,—1 Ns nN; nn, —1 
n,(ni — 1) 


(n, — 1) nj 
NiNs — 1 Nin, — 1 


o, — 1)" 


mula (9) then becomes 


which is K-R (21). The same substitution changes (8) to 


My n 


Tit = 


where V, is the same as in (2), but V., comes from (5). This for- 
mula may be put in terms of the same elements as the Kuder-Richard- 


where o,” is the sample variance of individual total test scores and 


If the test items have k possible weights, w;(i = 1, --- , k), and 
the total number of items, for all individuals, answered so as to have 


x 
a weight of w; is N; (where 5 N; = nin,;) , then we may express (7) as 


At this point we may compare this formulia with K-R (21). By 
substituting N,/nin, = p , N2/nin, = q, W: = 1, and w. = 0, we get 


For relatively large n, — 1, and consequently large n,n, — 1, we may 
substitute , and n,n, , respectively, with little loss of accuracy. For- 


: 
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Use of Formula (11) and Empirical Validation 


Formula (11) is convenient to use when the item counter on the 
I.B.M. Test Scoring Machine is not available, or is not desired in 
favor of a quicker estimate of reliability. If papers are to be scored 
by hand, it is only necessary to see that each paper has k values en- 
tered on it, each one representing the number of items answered so 
as to have the same weight, w;({=1,---, &). The sum of these values 
on each sheet is the total number of items in the test. The weighted 
sum of values on each sheet is the total score for that individual. If 
papers are to be scored by machine, the same values can be obtained 
by running the papers through the scoring machine fewer times than 
is necessary for a split-half estimate of reliability. Whenever items 
are marked right or wrong, and omits are treated as wrongs so that 
only two weights are possible, K-R (21) or a slight modification of 
it (if weights are other than one or zero) will suffice for a reliability 
estimate. If omits are weighted differently from rights or wrongs, 
formula (11) can be easily used if each test paper has the number of 
rights and the number of wrongs listed. The total number of omits 
can then be found from 


> Omits = nin, — Rights — Wrongs. (12) 


A common type of test with multiple weights for the indi- 
vidual items is the personality or attitude scale. In many of these 
tests each item has five choices expressing varying degrees of ap- 
proval, with weights of 1, 2, 3, 4, and 5 attached to these choices. A 
sample item is: 

The government should not meddle with private business. 


Choices: (1) Strongly (2) Disagree (3) Undecided 


disagree 
Weights: 1 2 3 
Choices: (4)Agree (5) Strongly 
agree 
Weights: 5 


Here each test paper need have no more than four values on it; how- 
ever, the use of all five values will serve as a check on the scoring. 
In a study a number of years ago,* there are available sufficient 
data for a comparison of the usual split-half reliability with Hoyt re- 
liability and with reliability based on formula (11) in the present 
paper. The tests used by Rundquist and Sletto are personality scales 


_ * Rundquist and Sletto. Personality in the depression. Minneapolis: Univ. 
Minn. Press, 1936. 
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in the areas of morale, inferiority, family, law, economics and educa- 
tion. Each of the scales has 22 items, and each item has five choices as 
in the illustration above. The necessary data are taken from table 19, 
page 45, and table 44, page 85. Reliabilities were computed separately 
for two groups, one of 500 males and one of 500 females. The re- 
sults are given in the table below. 


Comparison of Three Reliability Coefficients 


| 
500 Males | 500 Females 


| 

| 

| | 

Scale | 

| S-B | Hoyt |Formula| S-B | Hoyt | Formula 

| | | | | 
Morale .79 81 81 
Inferiority -78 .80 82 81 -76 
Family 83 -78 .74 .87 83 81 
Law .84 82 -78 .82 83 By i 
Economic Conservatism .80 .74 
Education 
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FACTORIAL DESIGN AND COVARIANCE IN THE STUDY OF 
INDIVIDUAL EDUCATIONAL DEVELOPMENT* 


PALMER O. JOHNSON AND FEI TSAO 
UNIVERSITY OF MINNESOTA 


This is the report of the application of the principles of factorial 
design to an investigation of individual educational development. The 
specific type of factorial design formulated was a 2 X 3 X 3 X 3 
arrangement, that is, the effect of sex, grade location, scholastic 
standing, and individual order, singly and in all possible combina- 
tions was studied in relation to educational development as meas- 
ured by the Iowa Tests of Educational Development. An application 
of the covariance method was introduced which resulted in increased 
precision of this type of experimental design by significantly reduc- 
ing experimental error. The two concomitant measures used to in- 
crease the sensitiveness of the experiment were initial status of 
individual development and mental age. Without these statistical 
controls all main effects and two first-order interactions would have 
been accepted as significant. With their use only sex (doubtful), 
scholastic standing, and individual order demonstrated significant 
effects. The chief beauty of the analysis of variance and covariance 
as an integral part of a self-contained experiment is demonstrated 
in the complete single analysis of the data. The statistical utiliza- 
tion of the experimental results has also been developed for purposes 
of estimation and prediction. The mathematical statistician is being 
continuously required to develop and analyze experimental designs 
of increasing complexity since the introduction of the analysis of 
variance and covariance. The mathematical formulation and solu- 
tion of the problem of this investigation is carried out. The methods 
illustrated and explained in this study, and modifications and ex- 
tensions of them are capable of very wide application. The general 
principles can be used to various degrees and in a number of ways. 


An earlier paper} has described in detail the use of factorial de- 
sign in an experiment in psychology. It is proposed to report here 
the application of the principles of factorial design and the appropri- 
ate statistical analysis to an investigation of individual educational 
development. While the experimental results are themselves impor- 
tant, we wish to develop and illustrate the procedures which supply 
a sufficient technique for collecting the primary data in such a man- 
ner that valid conclusions may be educed from them, and for extract- 
ing the relevant information contained in the data in the most effici- 
ent way. 

* For the research grant to finance this study, grateful acknowledgment is 


given to the Graduate School, the lg gg of Minnesota. — Se 
+ Johnson, Palmer O. and Tsao, Fei. Factorial design in the determination 


of differential limen values. Psychometrika, 1944, 9, 107-145. 
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The procedures are illustrated by original data which were col- 
lected in the University of Minnesota high school. The principal edu- 
cational factors chosen for study were sex, scholastic standing, and 
individual order of educational development. Other factors could 
have been included, but the above were considered particularly worthy 
of study and sufficient in number so as not to risk losing the attention 
of the reader unfamiliar with the technique in a maze of unavoidable 
details, as might happen were numerous factors included. 

It should be emphasized that the present work attempts to make 
available to the scientific reader the application of principles from 
the point of view of their utilization in the construction of more ef- 
ficient and comprehensive experimental designs and clearer, more 
concise, and more efficient statistical analysis of results. The situa- 
tion in this study is one of very wide occurrence. The traditional 
method for the study of the educational factors considered here would 
require a series of experiments each designed to test a single factor 
with arbitrary control of the other factors.* In modern designs the fac- 
tors are tested simultaneously, which makes it possible to explore the 
interactions as well as the main effects. The specific design developed 
for this study was a 2 X 3 X 3 X 3 factorial type, the two sexes, three 
grades, three scholastic standings, and three individual orders. The 
type of factorial design is of the kind where absolute replication is 
dispensed with and hidden replication is involved.+ This is the type 
desirable when large numbers of combinations are tested simultane- 
ously without repeated use of each combination. Most of the higher- 
order interactions are assumed to be statistically insignificant. All 
of the independent comparisons contained in the experiment are allo- 
cated to the factors tested and their interactions. Since there is no 
independent comparison ascribable to pure error, the highest-order 
interactions are used as a basis for measuring the precision of the 
main comparisons. 

A distinctive application to factorial design in this study is that 
of the analysis of covariance, which increases the precision of the 
experiment by use of the internal evidence of a self-contained experi- 
ment to reduce experimental error. By the precision of an experi- 
ment is meant the quantity of information it yields relevant to the 
hypothesis under test. In this study the concomitant measurements 
of two variables were used to increase the precision of the experi- 
mental comparisons. The statistical utilization of the findings has 
also been developed for purposes of estimation and prediction. 


* Ibid., p. 108. 
+ Fisher, R. A. The design of experiments, London: Oliver and Boyd, 1944, 


pp. 111-113. 
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The report is presented in two parts. Part I describes the de- 
sign and analysis of the investigation. The mechanization of the cal- 
culations required by the research worker for the analysis and inter- 
pretation of the data from modern experimental designs is a distinc- 
tive feature of Fisher’s contribution to the statistical craft. However, 
for the mathematical reader interested in the statistical model, Part 
II, which develops the mathematical formulation and solution of the 
problem, is presented. 


PART I 


The subjects in this investigation were students in the Univer- 
sity of Minnesota High School. The Jowa Tests of Educational De- 
velopment* were administered to all students in the ninth, tenth, 
and eleventh grades in September, 1942, and again in September, 
1943. The composite scores were used. This score is comprised of 
the scores on each of the following nine tests: 


Understanding of Basic Social Concepts 
Background in the Natural Sciences 
Correctness in Writing 

Ability to Do Quantitative Thinking 

Ability to Interpret Reading Materials in the 
Social Studies 

Ability to Interpret Reading Materials in the 
Natural Sciences 

Ability to Interpret Literary Materials 

General Vocabulary 

Uses of Sources of Information 


FPP Pr 


Standard scores are used in terms of a standard scale from 0 to 30 
with a mean of 15, determined from the combined grades. 

From the 168 students who had taken the two batteries of ex- 
aminations, 54 were selected, 18 each from the tenth, eleventh, and 
twelfth grades.+ The 18 individuals in the tenth grade were all of 
chronological age 15; those in the eleventh grade, 16; and those in 
the twelfth grade, 17. All mental ages were based on the scores from 
the Terman Group Test of Mental Ability. Since the intelligence tests 
were not administered to all students on the same date, mental ages 
for all were calculated as of the same date, i.e., May 21, 1939. Stu- 


* College of Education, State University of Iowa, Iowa City, Iowa, 1942. 

+ We designed the investigation so as to secure equal representation in the 
sub-classes. While this equality is not an essential condition, it is highly desir- 
able in order to avoid rather complicated mathematical formulation thus lead- 
ing to a more laborious statistical analysis. 


= a 


136 PSYCHOMETRIKA 


dents were assigned to scholastic groups on the basis of their honor- 
point ratios as follows: 


(1) Good —honor-point ratios equal to or 
greater than 2 

(2) Average—honor-point ratios between 1 and 2 

(3) Poor —honor-point ratios less than 1. 


Individual order of educational development was based upon the 
scores made on the second administration of the battery of examina- 
tions described above. The individual who attained the highest score 
was given rank 1, the individual next highest, rank 2, and the lowest 
score, rank 3. Where individuals were tied for rank the position was 
randomly determined. 
All samples were selected at random by the use of Fisher and 


TABLE 1* 
Classification of the Raw Data of This Study 


| Scholas- | Indi- | 
Sex | tie | vidual Grade 10 | Grade 11 | Grade 12 
| Standing | Order |Final Initial M.A. | Final Initial M.A. |Final Initial M.A. 
| | | | 
| ee! 30 =. 28 45 | 26 22 62 | 29 25 60 
| Good | 2 25 22 5B | 26 21 57 | 29 24 88 
| | 3 22 19 46 | 24 21 65 | 22 19 64 
| | | | 
56 | 24 25 54 | 28 21 64 
Male | Average | 2 38 55 <20 17 4T 
| 3 14 14 29 | 15 18 19 75 
| | | | 
| | 18 18 84 | 18 17 16 29 
8B 12 9 38 23 | 14 12 28 
| | | | 
| | 1 5 ee | 44 | 26 22 60 | 88 2 94 
| Good | 2 44 | 2 57 | 29 29 89 
| 3 6 | 2 19 62 | 26 22 78 
| | | 
| 20 «18 88 | 22 19 54. | 28 21 50 
Female | Average | 2 18 16 O88 57 
| | 14 14 47 16 17 43 
| | | | 
| 14 9 40° | 38 36 
| | 8 9 7 Bd ae. ae 48 | 10 9 14 


* The chronological age of every individual in grade 10 is 15 vears; grade 11, 16 years; grade 
12, 17 years. A hundred has been subtracted from Mental Age (in terms of months). Moreover, 
it has been assumed the mental test was administered on the 21st of May in 1939. 


i 
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Yates’ Random Sampling Numbers.* The primary data are shown 
grouped into the several sub-classes in Table 1. The sums of squares 
and of products for all sources of variation are recorded in Table 2. 


TABLE 2 
Sums of Squares and of Products for All Variations* 


Source of Variation df. Zy? 
Sex X Grade X Scholastic 


X Individual 8 16 26 690 13 21 —11 
Sex X Grade X Scholastic 4 25 35 942 29 130 135 
Sex X Grade X Individual 4 18 22 659 19 41 29 
Sex X Scholastic X Individual 4 5 9 515 2 10 19 
Grade X Scholastic X 

Individual 8 33 28 984 27 112 118 
Sex X Grade 2 32 53 638 40 138 160 
Sex X Scholastic 2 2 20 ia 8 6 —4 —14 
Sex X Individual 2 3 32 119 7 —14 —10 
Grade X Scholastic 4 37 29 1299 27 213 138 
Grade X Individual 4: 5 7 445 0 22 38 
Scholastic <X Individual 4 10 17 373 9 28 16 
Sex 1 30 19 69 24 45 36 
Grade 2 62 83 5837 70 594 697 
Scholastic 2 1087 824 9228 920 3073 2757 
Individual 74 292 220 1022 253 545 471 


Total 53 1607 1424 22831 1446 4954 4639 


* Where y = deviation score from the grand mean of final 
7; = deviation score from the grand mean of initial 
2 = deviation score from the grand mean of M.A. 
This will be true for all the following tables. 


The Test of Significance of Interactions and Main Effects 


We proceeded first to test the null hypothesis with respect to the 
interactions. Since no measure of pure error was available, the triple 
interaction, sex X grade X scholastic standing X individual, was used 
as error against which was compared each of the four second-order 
and each of the six first-order interactions} (Table 3). The null hy- 
pothesis was accepted for all interactions except two, sex < grade, and 
grade X scholastic standing. 

All of the interactions shown to be insignificant were pooled and 
constituted a mean square of 2.93, with 40 degrees of freedom for 


* Fisher, R. A., and Yates, F. Statistical tables. London: Oliver and Boyd, 
1938, pp. 82-87. 

+ It is customary to call the interaction involving two factors, an interaction 
of first order; to call the interaction involving three factors, an interaction of 
second order; etc. Sometimes for the sake of brevity, the terms single, double, 
and triple, interactions are used to denote the interactions of first order, second 
order, and third order, respectively. For instance, we may call the interaction 
sex X grade X scholastic individual, an interaction of third order. Or we may 


call it a triple interaction. Particularly in this study, this interaction may be re- 
garded as of highest order. 
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TABLE 3 


Test of Significance of Interactions 
(Considering final score only) 


Sumof Mean Hypothesis} 
Source of Variation d.f.° Squares Square F* Tested 

Sex X Grade X Scholastic 

X Individual (or Error) 8 16 2.00 — _— 
Sex X Grade X Scholastic 4 25 6.25 3.12 Accepted 
Sex X Grade X Individual | 4 18 4.50 2.25 Accepted 
Sex X Scholastic X Individual 4 5 1.25 — Accepted 
Grade X Scholastic X Individual 8 33 4.13 2.06 Accepted 
Sex X Grade 2 32 16.00 8.00 Remain in doubt 
Sex X Scholastic 2 2 1.00 — Accepted 
Sex X Individual vA 3 1.50 — Accepted 
Grade X Scholastic 4 37 9.25 4.63 Remain in doubt 
Grade X Individual 4 5 1.25 — Accepted 
Scholastic X Individual 4 10 2.50 1.25 Accepted 


Mean square of a tested variation 
* Where F = ——_-—___— 
Mean square of error or residual 

By referring to Snedecor’s tables of F (see Snedecor, G. W., Statistical methods applied to 
experiments in agriculture and biology. Iowa: Collegiate Press, 1938, pp. 174-77), we may use the 
following four rules in testing the hypothesis: (a) reject the hypothesis tested, if the calculated 
value of F is greater than the 1% point given in the tables; (b) accept the hypothesis tested, if the 
calculated value of F is less than the 5% point given in the tables; (c) remain in doubt, if the 
caleulated value of F lies between the 5% and 1% points given in the tables; (d) in the event 
that the mean square of error or residual is greater than the mean square of a tested variation, then 
we simply accept the hypothesis without calculating the value of F. 

These conventions are always true throughout the following tables. 

+ The hypothesis tested is a null hypothesis concerning the variation in the same row. For 
instance, the hypothesis regarding sex x grade is that there is no significant interaction between 
sex and grade. 


TABLE 4 


Complete Analysis of Variance 
(Considering final score only) 


Sum of Mean Hypothesis* 
Source of Variation df. Squares Square F Tested 
Residual 40 117 2.93 
Sex X Grade 2 82 16.00 5.46 Rejected 
Grade X Scholastic 4 37 9.25 3.16 Remain in doubt 
Sex 1 3 30.06 10.24 Rejected 
Grade 2 62 31.00 10.58 Rejected 
Scholastic 2 1037 518.50 176.96 Rejected 
Individual 2 292 146.00 49.83 Rejected 
Total 53 1607 


* The hypothesis tested is a null hypothesis regarding the variation in the same row. For in- 
stance, the hypothesis regarding grade is that there is no significant difference between grade-means. 
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testing the significance of the main effects and the doubtful inter- 
actions. The complete analysis of variance of the final scores is shown 
in Table 4. All of the main effects, sex, grade, scholastic standing, 
and individual order were found to be significant. One of the two 
previous doubtful interactions, sex < grade, became significant, the 
other, grade X scholastic standing, remained doubtful. 


Increasing the Precision by Concomitant Measurements 


Corrections were introduced into the comparisons, which made 
allowances for the variations in the initial scores on the examinations 
and in mental age scores. The first factor to be controlled statistically 
was the initial score. 

Influence of initial score. The necessary calculations for par- 
tialling out the effect of initial scores are given in Table 5. The theo- 
retical basis developing the mathematical formulas are given in Part 
II and procedures necessary for the calculation of the sum of squares 
and sum of products for each source of variation are summarized in 
Table 1, Part II. The test of significance of the interactions is given 
in Table 5. None of the interactions was found to be significant after 
the influence of initial scores had been allowed for. It was noted in 
Table 3 that the interactions, sex < grade and grade X scholastic 
standing, were in the region of doubt. 

The complete analysis of variance and covariance of the final 
scores, partialling out the effect of initial scores, is found in Table 6. 
It is to be noted that since none of the interactions was significant, 
46 degrees of freedom, reduced to 45 after adjustment, were available 


TABLE 6 


Complete Analysis of Variance and Covariance 
(Partialling out the effect of initial score) * 


Source of Adjusted Reduced 

Variation df. =y? = df.S.S. MS. F Hypothesist 
Residual 46 186 179 278 70.86 (45 70.86 157 — — 

Sex 1 30 8624 19 7.07 |1 6.39 639 4.07} Remain in doubt 
Grade 2 62 70 83 6.43 |2 6.75 3.38 2.15 Accepted 


Scholastic 2 1037 920 824 197.24 | 2 66.81 33.41 21.28 Rejected 
Individual 2 292 253 220 58.36 | 2 35.97 17.99 11.46 Rejected 


Total 53 1607 1446 1424 339.96 


Coefficients for 
Adjusting =y? 1 —1.28 .41 


* Where = .64 

+ Near the borderline of 5% significance and insignificance. 

} The hypothesis tested is a null hypothcsis concerning the variation in the same row. For 
instance, the hypothesis regarding sex is that there is no significant difference between sex-means 
when the effect of initial scores has been partialled out. 
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for the exact tests of significance.* From the tests of significance it 
was found that scholastic standing and individual order were signifi- 
cant and that sex differences were doubtful. The increased precision 
in the application of covariance is observed in comparing the results 
with those reported in Table 4 where five main effects and one inter- 
action were found to be significant and an additional interaction was 
in doubt. 

Influence of initial score and.mental age. We have now to con- 
sider the possible gain in precision which might result from eliminat- 
ing, or making allowance for, two variables simultaneously. The two 
concomitant measurements were initial score and mental age. The 
application of the covariance method to two independent variables in- 
volves the calculation of the sums of squares of the dependent vari- 
able and of each of the two independent variables, and the sums of 
products of each with the dependent variate to be adjusted, and with 
each other (see Table 7).¢ The calculations for the interactions of 
all orders are found in Table 7, and the significance of each source of 
variation has been determined after partialling out the effects of 
initial score and mental age. Since the adjustments for the two con- 
comitant variates have been obtained from the error term, two de- 
grees of freedom ascribed to error have been used in evaluating it. 
The reduced sum of squares ascribed to error is divided by the de- 
grees of freedom to obtain the mean square (1.41) appropriate to 
testing the significance of the remaining interactions. No significant 
interaction was found. There became available then 46 degrees of 
freedom before adjustment, reduced to 44 after adjustment, to test 
the significance of the main effects. 

The complete analysis of variance and covariance of the final 
scores, partialling out the joint effect of initial score and mental age, 
is presented in Table 8. A comparison may be made by referring 
again to Table 6 between the mean adjusted [Sy?]’s after partialling 
out the effect of initial score and mental age and those after par- 
tialling out the effect of initial score only. The changes brought about 
by considering the joint effect of initial score and mental age did not 
alter the conclusions drawn when only the effect of initial score was 
eliminated.t The final analysis, then, which has utilized all the evi- 


* See pp. 159-60, Part II, for explanation. 

+ See pp. 160-162, Part II, for the mathematical formulation of the procedures 
employed. 

t By ordinary methods, we obtain the following three zero-order correla- 
tions from the scores of all the individuals: 

— -96, 82, 61, 

where y denotes the final score; 1 denotes the initial score; and 2 denotes the 
mental age score. The first-order correlations with the third variable partialled 
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dence of the relevant data, led to the conclusion that there was a sig- 
nificant difference between the mean final scores of scholastic stand- 
ings and of the individual orders when adjustments were made for 
the differences in initial scores and in mental ages. Further obser- 
vations would appear to be necessary to establish whether or not a 
highly significant sex difference existed. The difference was signifi- 
cant at the 5 per cent level. 


Problems in Prediction and Estimation 

In the previous section it was established that a significant dif- 
ference existed among the means of final scores of the three scholastic 
standings and among the mean final scores of the individual orders. 
In this section we present the solution of the problems of estimating 
final scores from the scholastic standings and from the individual or- 
ders. There are two kinds of final scores, the observed and the pre- 
dicted. Therefore, two kinds of regression equations have been de- 
veloped, one of each for the scholastic standings and for the individual 
orders for the estimation of (1) observed final scores and (2) pre- 
dicted final scores.*- In other words, if one person has been classified 
in a certain individual order or in a certain scholastic standing what 
value of the observed and the predicted final score can be most prob- 
ably got without having any other information? These scores can be 
obtained from the regression equations developed.+ 

The estimation from scholastic standings. We present first the 
equations for the estimation of observed final scores from the differ- 
ent scholastic standings. First, it is necessary to obtain the predicted 
final scores from initial scores and mental ages for each scholastic 
standing. Here the multiple regression equation is 


y = + bor, = 51742, + 05192, . (1)t 


*In the equations that follow we should like to state here the distinction 
in notation used: ~ is used as the symbol of prediction and , of estimation. 

+ Errors of estimation can be calculated in the usual manner. 

t See page 160, Part II for the development of this equation. The values of 
b, and 6b, were calculated from the residual component (see Table 8). 


out are as follows: 


This shows that if we regard the final score as a dependent variable, then the 
effect of the initial score is much more significant than that of the mental age. 
So in our case, by removing only the effect of the initial score we can largely in- 
crease the precision of our results. However, it does not necessarily mean that 
we need not remove the effect of the mental age. So far as there is some rela- 
tionship between the mental age and the final score, it will always be advisable 
to use the technique of analysis of covariance to increase the precision of our 
results even if the increase is slight. 
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The necessary calculations are recorded in Table 9. 


TABLE 9 


Final Scores and Predicted Final Scores from Initial Scores 
and M.A. for Each Scholastic Standing* 


(Using y = b, + by == 5174 2, + .0519 


Good 25.2778 22.1111 59.3888 5.5000 4.6296 15.3383 3.1912 22.9690 
Average 19.5000 17.7778 45.8383 -.2778 .2963 1.2778  .2196 19.9974 
Poor 14.5556 12.5556 27.4444 —5.2222 4.9259 -16.6111 -3.4108 16.3670 


Total 19.7778 17.4815 44.0555 


* Where Y = mean final score, for instance, 25.2778 is the mean final score of all 18 individuals 
who have good standings. 


‘X; = mean initial score 

X2= mean mental age 

‘YY = mean deviation score from the grand mean of final score 
Ti = mean deviation score from the grand mean of initial score 


= mean deviation score from the grand mean of mental age 


predicted mean deviation score from the grand mean of final score 


“It elt 


icted mean final score 


- 19.7778, 


+ 19.7778. 


From the mean final scores, [Y2]’s, of different scholastic stand- 


ings we can obtain the estimated observed final scores, [Y]’s from 
scholastic standing by the following regression equations: + 


Linear form: Y = 19.7778 + 5.36112 ‘ (2) 
Quadratic form: Y = 19.5000 + 5.36112 + .41672?, (3) 


where x = 1,0, —1, which indicates good, average, and poor scho- 


lastic standings, respectively. 
From Table 10, it is noted that the variation due to scholastic 


+ The details of the method of calculating the regression equations are 
not described here. As we wish to use the analysis of variance for testing the 
goodness of fit, it is convenient to use orthogonal polynomials. The interested 
reader is recommended to refer to the following sources: 

Goulden, C. H. Methods of statistical analysis. New York: John Wiley, 
1939, pp. 219-246. 

‘Anderson, R. LL. and Houseman, E. E. Tables of orthogonal polynomial val- 
ues extended to N — 104. Agricultural Experiment Station, Iowa State College 
of Agriculture and “Mechanics Arts, Research Bulletin 297, 1942, pp. 595-606. 
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standing is quadratically significant at the 5% level. It was therefore 
necessary to develop equation (3) of quadratic form. From this equa- 
tion it is possible to estimate the observed final score from scholastic 


standing. 


Likewise, from the mean predicted final scores, [Y] ’s of different 
scholastic standings, we can obtain the regression equations for the 


estimation of predicted final scores, ty] ’s, from different scholastic 
standings 


A 


Linear form: Y = 19.7778 + 3.3010z, (4) 


A 


Quadratic form: Y = 19.9974 + 3.30102 — .32942°. (5) 


From equation (5) one can estimate the predicted final scores from 
scholastic standing. 

The summary of the findings for variation due to scholastic 
standing is given in Table 11. The arithmetical means on final scores 
for good, average, and poor scholastic standings, were 25.3, 19.5, and 
14.6, respectively. In Table 11, &(s) and &2(s) denote the linear 
and the quadratic polynomial coefficients for each scholastic stand- 
ing, respectively. These coefficients can be found in Fisher and Yates’ 
Table 23.* 

The sums of squares and of products for the two components of 
variation in Table 10 can be calculated from the figures given in Table 
11. For instance, to obtain Sy? linear (see Table 10), we have (i) to 


TABLE 11 
Summary of Results for Variation Due to Scholastic Standing 
Scholastic 
Standing a X, (8) 
Sums of Scores 455 898 1069 
Good 
Mean 25.2778 22.1111 59.3888 
Sums of Scores 351 820 816 
Average 0 —2 
Mean 19.5000 17.7778 45.3333 
Sums of Scores 262 226 494 
Poor —1 +1 
Mean 14.5556 12.5556 27.4444 


* Fisher, R. A., and Yates, F. Statistical tables. London: Oliver and Boyd, 
1988, pp. 54-59. 
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multiply each of three sums of Y scores (or mean Y scores) by the 
value é',(s) in the same row as itself; (ii) add up the three products, 
(iii) square the sum; (iv) divide by 18, which is the number of in- 
dividuals in each scholastic standing [if, in (i) we use the mean Y 
scores, then we multiply by 18 in this step]; and (v) divide by the 
sum of squares of &',(s), namely, 2. Following these steps, we have 


[445 (+1) + 351 (0) + 262 (—1)]? 
18 (2) 


= 1,035 


or 


[25.2778 (+1) + 19.5000 (0) + 14.5556 (—1)]?_ — 
2 


As another example, to obtain Sz,y quadratic (see Table 10), we 
have (i) to multiply each of the three sums of Y scores (or mean Y 
scores) by the value of &.(s) in the same row as itself and add 
these three products, (ii) multiply each of the three sums of X, 
scores (or mean X, scores) by the corresponding value of é’,(s) and 
add up the three products; (iii) find the products of the above two 
sums of products; (iv) divide by 18 [if in (i) we use the mean 
scores, then we multiply by 18] ; and (v) divide by the sum of squares 
of &’,(s), namely, 6. Following these steps, we have 


{[455 (+1) + 351 (—2) + 262 (+1)][89 (41) 
+ 320 (—2) + 226 (+1)]}/18 (6) = —-2 


or 


{ [25.2778 (+1) + 19.5000 (—2) + 14.5556 (+1) ][22.1111(+1) 
+ 17.778 (—2) + 12.5556 (+1)] (148)}/6=-2. 


The estimation from individual order. We now present the equa- 
tion for the estimation of observed final scores from different indi- 
vidual orders. In Table 12 are tabulated the final scores, and pre- 
dicted final scores from initial scores and mental ages for each indi- 
vidual order. The following regression equations were established 
for estimating the observed final scores from individual orders: 


Y = 19.7778 + 2.8334 x, (6) 


where x = 1,0, —1, which denotes first, second, and third order, re- 
spectively. Only the linear component of variation due to individual 
order was found to be significant (see Table 13). 
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TABLE 12 


Final Scores and Predicted Final Scores from Initial Scores 
and M.A. for Each Individual Order* 


(Using y = .5174 2, + .0519 z,) 


~ ~ 


1 22.4445 19.8889 48.7777 2.6667 2.4074 4.7222 1.4907 21.2685 
2 20.1111 17.6111 45.1111 8383 .1296 1.0556 .1218 19.8996 
3 16.7778 14.9445 38.2777 -3.0000 -2.5370 -5.7778 -1.6125 18.1653 
Total 19.7778 17.4815 44.0555 


* Where all the notations are the same as in Table 9. 


Finally, we present the equation for the estimation of predicted 
final scores from the different individual orders. Following the same 
procedures as before we obtained the linear equation 


A 
~ 


Y = 19.7778 + 1.5516 x, (7) 


where x has the same meaning as in (6). 

The summary of the findings for variation due to individual order 
is given in Table 14. The mean final scores were 22.4, 20.1, and 16.8 
for individual orders 1, 2, and 8, respectively. Their respective means 
on the initial examinations were 19.9, 17.6, and 14.9, and of the men- 
tal ages in months, 148.8, 145.1, and 138.3. In Table 14 &',(i) and 
&',(2) denote the linear and the quadratic polynomial coefficients for 
each individual order, respectively. The sums of squares and of prod- 
ucts for the two components in Table 13 can be calculated from the 
figures given in Table 14 in the same way as indicated in the previous 
section. 


Summary and Conclusions 


1. The principles of factorial design were applied in this investiga- 
tion of individual educational development. The specific type of 
factorial design used was a 2 X 3 X 8 X 8 arrangement, that is, 
the two sexes, three grades, three scholastic standings, and three 
individual orders. 

2. The appropriate statistical analysis for this type of design is the 
analysis of variance. An application of the covarience method was 
introduced into the method of analysis to control two sources of 
variation, the mental age and the initial scores of the experi- 
mental subjects. In this way, the precision of the experimental 
comparisons was increased by reducing the experimental error. 
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TABLE 14 
Summary of Results for Variation Due to Individual Order 
Individual XxX, &’, (2) 
Sums of Scores 404 858 878 
First +1 +1 
Mean 22.4445 19.8889 48.7777 
Sums of Scores 862 317 812 
Second 0 —2 
Mean 20.1111 17.6111 45.1111 
Sums of Scores 302 269 689 
Third —1 +1 
Mean 16.7778 14.9445 38.2777 


3. When final score results alone were considered the following com- 
ponents were found to be significant: 


Main effects Interactions 
X Sex X Grade 
Grade Grade X Scholastic Standing (doubtful) 


Scholastic standing 
Individual order 


4, After the effect of initial score and mental age was partialled out, 
all the interactions were insignificant and there was no significant 
difference among grades. The only factors which remained sig- 
nificant were: 


Sex (doubtful) 
Scholastic standing 
Individual order 


5. Regression equations were developed to estimate the observed and 
the predicted final scores from classifications of scholastic stand- 
ings and from individual orders of educational development. The 
method was used of fitting by orthogonal polynomials. The analy- 
sis of variance was used to test the goodness of graduation. 


The methods and procedures illustrated in this investigation are 
capable of extensive application. The conventional methods for at- 
tacking problems of this kind are usually inexact, of low precision, 
and incapable of yielding conclusive results. The introduction of co- 
variance methods into factorial design will likely prove to be very 
useful, if not indispensable, in many psychological and educational 
problem. 

It is customary to consider variation within sub-groups as resi- 
dual, or experimental error. The results from this study point to the 
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fact that individual differences within a group are so strikingly sig- 
nificant that we cannot regard them as a residual. Where experi- 
mental designs take this factor into account, increase in precision may 
be expected, as has been convincingly demonstrated in this investiga- 
tion. 

The mathematical formulation and the solution underlying the 
method and analysis applied in Part I follow. 


PART II 
The derivation of sums of squares for each source of variation 
have been shown in the previous paper.* It is not necessary that it 
be repeated here. When the analysis of covariance technique is used, 
we have to calculate the sum of products. Let us consider only the 
simplest way of calculating sum of products. Suppose we define 


X;= the raw score of the i-th individual on X variable 
Y;,= the raw score of the i-th individual on Y variable 

a; = the deviation from the mean X for the i-th individual 
y; = the deviation from the mean Y for the i-th individual 
$ =1,2,-:- N 

N =the number of cases 


We have no difficulty in getting 


(= Xi)’ 
N 
N 
(= Xi) (2 


It is customary that the second term of the right-hand member 
of each equation mentioned above is called the “correction” term. On 
the other hand, the first term is only the straightforward calculation 
of the sum of squares (or products) of all the raw scores. If we al- 
ways keep in mind what the difference is between the “correction” 
term for the sum of squares and that for the sum of products, then 
we can get all the solutions. 


* Johnson and Tsao, op. cit., pp. 128-148. 


PALMER 0. JOHNSON AND FEI TSAO 153 


As for our problem, we define 


Y.ij¢ = the final standard score of the ¢-th individual of the 
j-th scholastic standing in the i-th grade and the s-th sex, 


ies, = the initial standard score of the t-th individual of 
the j-th scholastic standing in the i-th grade and the s-th sex, 
Xe, = the mental age score of the ¢th individual of the 
j-th ehuciakies standing in the i-th grade and the s-th sex, 
where s = 1, 2;1= 1, 2,3; j= 1, 2,3; 1, 2, 3. 
Assume that 


Y,,,,)3 a; 


sijt 


sijt 
sijt 
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18 18 
ds, = ; 
18 
dy. = dy => , 
4 27 | 18 
27 18 
18 
27 18 
18 18 
5A 54 5A 
5A 54 
54 


Then all the calculations of the sum of squares and sum of prod- 
ducts for each source of variation will be summarized in Table 1. 
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Now we consider how to eliminate the effect of initial scores in 
our study. By using regression of y on « , we have 


y = br, (2) 


where y denotes the predicted deviation from the mean of final scores, 
x, denotes the deviation from the mean of initial scores, and b is re- 
gression coefficient. 

Minimizing the difference between the actual y score and the 
predicted y score, we have 


r= TY — bz,)*. (3) 
Differentiating y? with respect to b , we obtain 
(4) 


Consequently, we get 
= — bz,)? = Sy? — 2b Say + 6? (5) 
So the coefficients for adjustying Sy? for all sources of variation 


are as follows: 
Sry >2:? 
1 —2b b? 


Remembering that the value of 6 comes from the error term 
(when we make the test of significance of interactions) or the resi- 
dual (when we make a complete analysis), let us consider this term 
only. By simple algebraic manipulation, we have 


= Dy — + b? 


. (6) 
(Sxy) 
~ 
Therefore the degrees of freedom in error or residual will be dimin- 
(Sry)? 


2 
1 


ished by 1 because of the restriction of b},y or a in making 
adjustment. 

The value of b used for adjusting Sy? for all sources of variation 
is a statistical estimate subject to errors of random sampling. So it 
is advisable in making an exact test of significance that we use the 
reduced value* instead of the adjusted value. For instance, in Table 


_ ™* Fisher, R. A. Statistical methods for research workers, (7th edition). Lon- 
don: Oliver and Boyd, 1938, pp. 281-294. 
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6 of Part I, the reduced value of the sum of squares for the variation 
of sex is obtained by the following procedure: 

Taking those parts of Table 6 of Part I which refer to residual 
and sex only, we obtain the reduced values of Sy?, respectively, for 
the residual and for the total, by deducting from Sy? in each case 
the quantity 

(Sry)? 


derived from the same line. This gives for the residual the reduced 
value of Sy’, 70.86, as in Table 6 of Part I. For the total we have 
216 — 188.75 = 77.25, 


corresponding to 46 degrees of freedom. Subtracting the reduced Sy? 
for residual from the reduced Sy? for total, we find the reduced Sy? 
ascribable to the 1 degreé of freedom for sex to be 6.39, which is the 
value to be compared with the reduced Sy? for residual, in making 
an exact test. The whole procedure is shown in Table 2. 


TABLE 2 
Illustration of Test of Significance with Reduced 2y? 
(Partialling out the effect of X,) 


Source of Reduced 
Variation df. 2y? at. SS: Bs. Hypothesis 
Residual 46 186 179 278 45 70.86 1.57 _ 


Sex 1 30 24 19 1 639 6.389 4.07 Remainin 
doubt 


Total 47 216 203 297 46 17.25 


Finally, we work out the regression coefficients for adjusting 
Sy’ in partialling out both the effects of initial scores and M.A. By 
using the regression of y on x, and x, , we have 


y = b,x, + ber, 


where y denotes the predicted deviation final score, x, denotes the 
deviation initial score, x, denotes the deviation M. A., b, is the partial 
regression coefficient of x, , and b, is the partial regression coefficient 
of 

Minimizing the difference between the actual y score and the pre- 
dicted y score, we have 


— biz, — bex2)?. (7) 
Differentiating 7’ with respect to b, and b. , we have 


~ 
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b, Sait, + b, = Srey. 
By using the determinant method, we have 
— 
— 
Consequently, we get 
P= Ty — — bet2)? = Ty? + b,? 
(10) 


+ b.? >z,* 2b, 2b. + 2b, b, 


So the coefficients for adjusting Sy for all sources of variation are 
as follows: 


1 b,? b.? —2b, —2b. 2b,b. 


The values of b, and b. come from the error term or the residual 
term. Let us consider this term only; we have, by troublesome alge- 
braic manipulation, 


C= Dy + Sx? + b.? — 2b, 
2b. 2b, b. (11) 
= — b, Sry — b. Sry. 


Therefore, the degrees of freedom in error or residual will be dimin- 
ished by 2 because of the two restrictions of b;Sx,y and b.Sx.y in 
making adjustment. 

Once more in making an exact test of significance for each source 
of variation we use the reduced Sy? instead of the adjusted Sy’. The 
calculation of the reduced Sy? is similar to the one used earlier. To 
give an example, in Table 8 of Part I, the reduced Sy? for the varia- 
tion of grade is obtained by the following process: 

Taking those parts of Table 8 of Part I which refer to residual 
and grade only, we obtain the reduced Sy*, respectively, for the resi- 
dual and for the total, by deducting from Sy’ in each case the quan- 
tity 

Dx? (Sry)? — Sry 4 — Say S722 
Da? — La? (Lave)? 


or 


| 
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(S2,y)? — 2 Say Sry + (Sry)? 
— 


derived from the same line. This gives the reduced Sy? for residual, 
57.21, as in Table 8 of Part I. For the total we have 


248 — 87.90 = 60.10, 


corresponding to 46 degrees of freedom. Subtracting 57.21 from 
60.10, we find the reduced Sy? ascribable to the 2 degrees of freedom 
for grade to be 2.89. The whole procedure of making an exact test 
of significance is summarized in Table 3: 


TABLE 3 
Illustrations of Test of Significance with Reduced =y2 
(Partialling out the effects of X, and X,) 


Source of | | i | | | Reduced 

Variation df. | | | SS. MS. F.  Hypothes 
Residual 46 | 186 | 278 | 6675| 179 | 697 | 678 |44 5721 130 — — | 
Grade 2 | 62| 83 | 5837| 70| 594| 697|2 289 145 112 Accepted 
Total 48 | 248 | 261 |12512| 249 | 1291 | 1375 [46 60.10 


; 
‘ 
q 
q 


ERRATUM 


Dr. Frederick B. Davis points out that the 
minus sign should have been a plus sign in for- 
mula (6), p. 59, of his article, “The Reliability of 
Component Scores,” in Volume 10, No. 1 of this 
journal. 
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